High CPU Idle Usage with pfsense/OpenBSD

dennisd

New Member
Oct 12, 2013
3
0
1
Hey all,
since I've updated on the latest packages (nonStable/free Repo) I did notice a cpu increase/utilisation for my KVM pfsense/freebsd machine.

Does anybody else has the same problem? Is there a workaround?

Thanks
 

nixcz

New Member
Sep 16, 2014
11
0
1
Yes, I happened to upgrade right at the release of 3.3 (great work, and thanks to the devs), but ran into this problem. I use pfSense and other FreeBSD VMs heavily and was dismayed to see idle VMs hit 10-15% of two allocated CPU cores.

For me, this only seems to affect FreeBSD version 8. Current pfSense 2.1.5 is based on FreeBSD 8.3 so it's affected. I have an 8.x VM that has this problem, and booting the 8.4 release ISO also shows it. I booted the 9.1, 9.3, 10.0, and 10.1-BETA release ISOs, and they seem to work just fine and do NOT show the problem. I only tried amd64; have no idea how i386 behaves.

Note that disabling the tablet pointer in the VM options tab seems to reduce CPU usage by a noticeable 2-3% regardless of FreeBSD version, so that's low hanging fruit to hit first.

After a lot of trial and error, it seems like we have to resort to some "old" tricks like setting kern.hz to 100 instead of the default 1000. I added the following in my /boot/loader.conf.local (in addition to the virtio lines) on one of my pfSense VMs and will run it for a while before upgrading other hosts to Proxmox 3.3.

Code:
hint.apic.0.clock=0
kern.hz=100
You can find some other options at the bottom of this well known thread. Up to you to test and see what works best.

https://communities.vmware.com/thread/87794

Note that changing the CPU type, number of cores, virtio vs. non-virtio devices, etc. didn't seem to have much effect for me. YMMV.

And finally, I don't mean this as a complaint, but as a general observation, that FBSD support on Proxmox/KVM has been a lot more fragile than I would like to see for production systems. I count at least three major problems in the last ~two years that have been addressed (or not) in varying amounts of time.

There was that SeaBIOS issue, which had a published workaround within a day, the Ivy Bridge/Haswell CPU issue, which AFAIK still requires using a non-default kernel (read a report that the 3.10 non-OpenVZ kernel fixes this?), and now this idle CPU regression.
 

nixcz

New Member
Sep 16, 2014
11
0
1
Hi mir, can you share some details about your setup?

Physical hardware platform, pfSense version and i386 or amd64, any special tuning parameters, did you upgrade to Proxmox 3.3 from the no-sub repo, etc?

I'm seeing the problem on a Nehalem class server (Supermicro motherboard). I can get idle CPU under 4% only with the tuning I mentioned before.

Booting up the 8.4 release ISO (now tested both i386 and amd64), the VM idles at a 13-14% CPU. This is a minimal test VM with 512MB RAM, 1 CPU core (type host), one CD drive for the ISO image, no hard drives and no NICs.

pveversion: proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
 

mir

Well-Known Member
Apr 14, 2012
3,483
97
48
Copenhagen, Denmark
Physical hardware platform: AMD Opteron on Supermicro motherboard
pfSense: 2.1.5-RELEASE (i386) FreeBSD 8.3-RELEASE-p16
Hardware: 1 5GB disk virtio. 4 nics E1000. 512MB/1.00GB memory using baloon driver. 2 sockets (qemu32)
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
 

nixcz

New Member
Sep 16, 2014
11
0
1
Thanks for the info. Since you're on Opteron, I'm not sure if we can make any meaningful comparisons. I haven't used any of those in 7-8 years. Would love to know what other people are seeing.
 

nixcz

New Member
Sep 16, 2014
11
0
1
Here's something interesting. The increased CPU usage can be seen inside the VM as interrupt and system time. It's not purely on the host side. People with this problem might be able to see it reflected in the System tab of pfSense's RRD graphs page. You'll notice the blue and yellow shaded portions are noticeably higher than before the upgrade.

By now I've upgraded several more systems.

On a DL380g5 host with Xeon 5160 CPUs, I have a pfSense system configured with 1 socket, 2 cores, CPU type host, 512MB/2GB RAM, a SCSI disk and 4 virtio NICs. The firewall system is not loaded and and only has about 0.5 mbits/sec of traffic going through it. It runs OpenBGP, but this is not very active, announcing only a single prefix and not really taking any inbound routes.

As seen on the VM side: Before the upgrade, total CPU time was around 5% (basically all blue, system time), and interrupt time was not visible on the graph. After the upgrade, system time is around 10% and interrupt time is around 5%, so the total hovers around 15% with spikes to 20+%.

On the host side, this system used to be at 15% of 2 CPUs, but is now at 35-40%.

On newer physical hosts (like a Xeon X5570 server), the CPU time may be hard to see because the RRD graph is basically at 0. But the increase in system and interrupt time is still visible as thicker blue and yellow coloration at the bottom of the graph.
 
Last edited:

mir

Well-Known Member
Apr 14, 2012
3,483
97
48
Copenhagen, Denmark
I have been able to make virtio-net work:)
The trick was to Disable hardware checksum offload found under System->Advanced->Networking.

These two is also disabled (I can't remember if this is the default setting)
- Hardware TCP Segmentation Offloading
- Hardware Large Receive Offloading
 

nixcz

New Member
Sep 16, 2014
11
0
1
I have been able to make virtio-net work:)
The trick was to Disable hardware checksum offload found under System->Advanced->Networking.

These two is also disabled (I can't remember if this is the default setting)
- Hardware TCP Segmentation Offloading
- Hardware Large Receive Offloading
Sorry, didn't know you had a problem with virtio NICs, or I would have mentioned the fix. (Some people prefer not to use virtio for various reasons, like I went with a scsi drive on that VM mentioned above.) That's a known issue. I believe the other two were turned off by default recently. I remember being surprised to see them set on a new install when I went in specifically to turn off the first option for use with virtio.

But that doesn't seem to be relevant to the high CPU utilization...

I am, however, starting to wonder if virtio exacerbates the high CPU load and increases interrupt time instead of decrease it. Haven't had a chance to run more in depth testing. That install I mentioned earlier is running at almost double the normal CPU and has me worried. It's one of my older systems so higher utilization tends to show up more dramatically. But really shouldn't run at 40% CPU just to pass half a megabit of traffic. ;^(
 

vmalaga

New Member
Jul 3, 2012
4
0
1
I have the same issue with the last 3.3 update and now our pfsense fw is allways with 50% CPU and with the params at the loader.conf.local now is 20% CPU usage.

pfsense 2.1.3-RELEASE (amd64)
FreeBSD 8.3-RELEASE-p16

we are also using the 3.10 kernel because the Ivy Bridge/Haswell CPU issue,
 

nixcz

New Member
Sep 16, 2014
11
0
1
I have the same issue with the last 3.3 update and now our pfsense fw is allways with 50% CPU and with the params at the loader.conf.local now is 20% CPU usage.

pfsense 2.1.3-RELEASE (amd64)
FreeBSD 8.3-RELEASE-p16

we are also using the 3.10 kernel because the Ivy Bridge/Haswell CPU issue,
FYI, I have some E3-1270 v3 systems running Proxmox 3.3 with the default 2.6.32-33-pve kernel. Not positive, but I think that 64bit FBSD on Haswell problem might have been resolved in the new QEMU package, in case you're not totally comfortable running non-default kernels.

A pfSense VM on a host with this CPU can idle at 2-3% typically (with selected loader tweaks above), with bursty utilization when actively passing traffic.

A pfSense VM on an HP DL360 G5 with Xeon 5160 CPUs will idle at 25-30% CPU even with the tweaks.
 

vmalaga

New Member
Jul 3, 2012
4
0
1
FYI, I have some E3-1270 v3 systems running Proxmox 3.3 with the default 2.6.32-33-pve kernel. Not positive, but I think that 64bit FBSD on Haswell problem might have been resolved in the new QEMU package, in case you're not totally comfortable running non-default kernels.

A pfSense VM on a host with this CPU can idle at 2-3% typically (with selected loader tweaks above), with bursty utilization when actively passing traffic.

A pfSense VM on an HP DL360 G5 with Xeon 5160 CPUs will idle at 25-30% CPU even with the tweaks.
Our servers are HP DL360 G8 with two E5-2660 v2 and the 64bits FBSD pfsense hangs on startup with default kernel.
Until yesterday I was comfortable with the 3.10, but now with this CPU Load, not so much :mad:

you can see the increase on CPU load on the virtual pfsense under proxmox

pfs01_load.png
 

mir

Well-Known Member
Apr 14, 2012
3,483
97
48
Copenhagen, Denmark
For default kernel you must use qemu64 as processor.

As mentioned above adding this to /boot/loader.conf.local
hint.apic.0.clock=0
kern.hz=100

Reduces load considerably when using virtio.
 

vmalaga

New Member
Jul 3, 2012
4
0
1
For default kernel you must use qemu64 as processor.

As mentioned above adding this to /boot/loader.conf.local
hint.apic.0.clock=0
kern.hz=100

Reduces load considerably when using virtio.
Yes mir, i have add this params to loader.conf.local but still the load goes from 20% to 55%.
I think that this params are more focused for idle CPU functions, but my pfsense is very active, fw, haproxy and snort

And like say nixcz the interrupts inside pfsense now are really visible just after the upgrade, this is a very bad notice and unacceptable regresion


pfsense_interrupts.png
 

oxygen8.ireland

New Member
Mar 2, 2015
1
0
1
Just to confirm seeing this in Proxmox 3.4 with pfsense 2.1.5. Also confirmed adding kern.hz=100 to /boot/loader.conf.local with reboot and turning "Use Tablet for Pointer" off reduced idle CPU usage from approx 16% to %4 (graph attached).

Two articles I found interesting on this :

https://blog.pfsense.org/?p=115
http://serverfault.com/questions/422790/does-kern-hz-still-have-any-relevance-in-freebsd-if-dynamic-tick-mode-is-enabl

Looks like pfsense 2.2 might sort this being based on freebsd 10.1


---------------------------------------
VM Spec :
---------------------------------------
CPU : 4 x CPU Cores (kvm64) (based on physical Xeon E5640 @ 2.67Ghz - duel cpu)
Memory : 4GB
Network Device : e1000 x 7
---------------------------------------


---------------------------------------
Software :
---------------------------------------
Proxmox Version : 3.4
Pfsense : 2.1.5


proxmox_cpu_usage_pfsense.png
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!