[SOLVED] Weird issue when virtualizing pfSense on Proxmox VE

[TomSawyer]

I decided to try and virtualize pfsense because sometimes you don't have a spare bare metal hardware to dedicate to it or you do but it's just not worth it for the little amount of resources that pfsense needs (in small to medium networks).

Everything works fine except for one weird issue that I've noticed and I'm pretty sure didn't exist when it was on bare metal.

The issue is this: When I run a speed test (Speedtest.com or others) after a few seconds it starts affecting the response time (RTT and RTTsd) after a few seconds it starts affecting the response time (RTT and RTTsd) of all the gateways (VPN and WAN) raising the latency to a point where I think they would go down if I were to download/upload on full speed for more than a minute or two.

After a few tests I've noticed it's actually the upload part that chokes these gateways.

As I said, I'm pretty sure it didn't exist before. Otherwise I would have noticed such behavior and connections would be going down every few minutes...I think.

I set this VM to have all the CPU cores (8), but perhaps on bare metal every VPN client got its own thread and with virtualization it shares these threads with everything that is on proxmox (common sense) ? Although there's nothing on proxmox right now except for that pfSense VM.

Any idea why does it happen?

Thank you,

 

[aaron]

Did you disable hardware offloading? https://docs.netgate.com/pfsense/en...-pfsense-software-to-work-with-proxmox-virtio

 

[TomSawyer]

Yes, I did. And I would feel if the webgui was sluggish as they say in the documentation.

 

[keropiko]

I have a similar problem as posted here

https://forum.proxmox.com/threads/latency-and-packet-loss-issue.70141/

.
With pppoe and opnsense when i run speedtest or high bandwidth is beeing at my network used, i suffer packet loss and latency to the main pppoe gateway and the vpns too.
I have also disabled offloading on opnsense. I don't know if i have to disable also to all the cards i have at the server running proxmox through proxmox network configuration. I haven't tried yet in order to avoid network problems.

 

[TomSawyer]

I'm moving back to a bare metal (mini itx case) pfSense install. Not just because of the issue described here but also because of many other issues that come with virtualizing whatever runs/manges/protects your entire infrastructure/network. It makes sense only if your entire enviroment is virtualized or in the cloud etc. For a local physical one, the main gateway should standalone imo. I don't mind virtualizing everything else, including a NAS

I only wish there was a sff cheap hardware exactly for use cases like these. For example - An average CPU with AES-NI capabilities and more than one core + 2-8GB of an average speed RAM + 2-3 Ethernet ports, not bigger than a 3.5" HDD case.
In an ideal fair sane world such hardware should cost not more than 50-70$, maybe a 100$ at max. My humble opinion only...

Reporting back: Even after switching back to a bare metal setup the problem didn't go away. Seems to be an issue with pfSense 2.4.5-p1...maybe it happens when using VPN connections/gateways...idk I'm still waiting for an answer on Netgate's forum.
Maybe be ISP related? Some sort of limitations etc. or PPPoE specific issue idk...

Edit: An answer from a Netgate employee - "That's not unusual when the connection is congested. You're probably hitting some bad buffer bloat somewhere. You may be able to improve that with some traffic shaping in pfSense. Or you can just accept that's how your WANs behave and tune the monitoring parameters accordingly."

 

[aaron]

I'm moving back to a bare metal (mini itx case) pfSense install. Not just because of the issue described here but also because of many other issues that come with virtualizing whatever runs/manges/protects your entire infrastructure/network. It makes sense only if your entire enviroment is virtualized or in the cloud etc. For a local physical one, the main gateway should standalone imo. I don't mind virtualizing everything else, including a NAS

I only wish there was a sff cheap hardware exactly for use cases like these. For example - An average CPU with AES-NI capabilities and more than one core + 2-8GB of an average speed RAM + 2-3 Ethernet ports, not bigger than a 3.5" HDD case.
In an ideal fair sane world such hardware should cost not more than 50-70$, maybe a 100$ at max. My humble opinion only...

I agree with that. There are quite a few options out there for small x86 based machines with at least 2 NICs like the PC Engines APUs, Fitlet2 or other machines if you search. The problem is usually the price point because these are rather niche products. You are usually in the 150 to 300 $/€ range.

 

[spektrolyte]

Have you looked at http://espressobin.net/? Might be the hardware you're looking for.

 

[TomSawyer]

Have you looked at http://espressobin.net/? Might be the hardware you're looking for.

Thank you I checked prices and if you want 2GB RAM it would cost more than 75$. For that price you can buy a good Motherboard-CPU combo add a little extra money and have a machine that is either a monster Firewall or a mini server that acts as a firewall, NAS (basic) and other services or micro services etc. But when there's just no need for more than 1GB RAM or when a very small (physically) device is needed than yeah, this espressobin looks good.

 

[TomSawyer]

Updating about the latency issue: After a few tests and a vibrant post on the pfSense forum, it turns out it has nothing to do with proxmox, virtualization or pfSense itself. It's the asymmetric DSL PPPoE connection and the ISP. Got the same result (high upload "loaded" value) on fast.com when I connected a PC directly to the ISP's equipment:


The solution the members on the pfSense forum recommend is using traffic-shaping and/or fq-codel, which is something I'll try another day.

 

[aaron]

Have you looked at http://espressobin.net/? Might be the hardware you're looking for.

While they are definitely interesting boards, they are ARM based AFAICT and not X86.

Updating about the latency issue:

Thanks for the update You can mark the thread as solved by selecting the prefix in the dropdown next to the title field.