High CPU Idle Usage with pfsense/OpenBSD
- Thread starter dennisd
- Start date
Yes, I happened to upgrade right at the release of 3.3 (great work, and thanks to the devs), but ran into this problem. I use pfSense and other FreeBSD VMs heavily and was dismayed to see idle VMs hit 10-15% of two allocated CPU cores.
For me, this only seems to affect FreeBSD version 8. Current pfSense 2.1.5 is based on FreeBSD 8.3 so it's affected. I have an 8.x VM that has this problem, and booting the 8.4 release ISO also shows it. I booted the 9.1, 9.3, 10.0, and 10.1-BETA release ISOs, and they seem to work just fine and do NOT show the problem. I only tried amd64; have no idea how i386 behaves.
Note that disabling the tablet pointer in the VM options tab seems to reduce CPU usage by a noticeable 2-3% regardless of FreeBSD version, so that's low hanging fruit to hit first.
After a lot of trial and error, it seems like we have to resort to some "old" tricks like setting kern.hz to 100 instead of the default 1000. I added the following in my /boot/loader.conf.local (in addition to the virtio lines) on one of my pfSense VMs and will run it for a while before upgrading other hosts to Proxmox 3.3.
You can find some other options at the bottom of this well known thread. Up to you to test and see what works best.
https://communities.vmware.com/thread/87794
Note that changing the CPU type, number of cores, virtio vs. non-virtio devices, etc. didn't seem to have much effect for me. YMMV.
And finally, I don't mean this as a complaint, but as a general observation, that FBSD support on Proxmox/KVM has been a lot more fragile than I would like to see for production systems. I count at least three major problems in the last ~two years that have been addressed (or not) in varying amounts of time.
There was that SeaBIOS issue, which had a published workaround within a day, the Ivy Bridge/Haswell CPU issue, which AFAIK still requires using a non-default kernel (read a report that the 3.10 non-OpenVZ kernel fixes this?), and now this idle CPU regression.
For me, this only seems to affect FreeBSD version 8. Current pfSense 2.1.5 is based on FreeBSD 8.3 so it's affected. I have an 8.x VM that has this problem, and booting the 8.4 release ISO also shows it. I booted the 9.1, 9.3, 10.0, and 10.1-BETA release ISOs, and they seem to work just fine and do NOT show the problem. I only tried amd64; have no idea how i386 behaves.
Note that disabling the tablet pointer in the VM options tab seems to reduce CPU usage by a noticeable 2-3% regardless of FreeBSD version, so that's low hanging fruit to hit first.
After a lot of trial and error, it seems like we have to resort to some "old" tricks like setting kern.hz to 100 instead of the default 1000. I added the following in my /boot/loader.conf.local (in addition to the virtio lines) on one of my pfSense VMs and will run it for a while before upgrading other hosts to Proxmox 3.3.
Code:
hint.apic.0.clock=0
kern.hz=100You can find some other options at the bottom of this well known thread. Up to you to test and see what works best.
https://communities.vmware.com/thread/87794
Note that changing the CPU type, number of cores, virtio vs. non-virtio devices, etc. didn't seem to have much effect for me. YMMV.
And finally, I don't mean this as a complaint, but as a general observation, that FBSD support on Proxmox/KVM has been a lot more fragile than I would like to see for production systems. I count at least three major problems in the last ~two years that have been addressed (or not) in varying amounts of time.
There was that SeaBIOS issue, which had a published workaround within a day, the Ivy Bridge/Haswell CPU issue, which AFAIK still requires using a non-default kernel (read a report that the 3.10 non-OpenVZ kernel fixes this?), and now this idle CPU regression.
Hi mir, can you share some details about your setup?
Physical hardware platform, pfSense version and i386 or amd64, any special tuning parameters, did you upgrade to Proxmox 3.3 from the no-sub repo, etc?
I'm seeing the problem on a Nehalem class server (Supermicro motherboard). I can get idle CPU under 4% only with the tuning I mentioned before.
Booting up the 8.4 release ISO (now tested both i386 and amd64), the VM idles at a 13-14% CPU. This is a minimal test VM with 512MB RAM, 1 CPU core (type host), one CD drive for the ISO image, no hard drives and no NICs.
pveversion: proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
Physical hardware platform, pfSense version and i386 or amd64, any special tuning parameters, did you upgrade to Proxmox 3.3 from the no-sub repo, etc?
I'm seeing the problem on a Nehalem class server (Supermicro motherboard). I can get idle CPU under 4% only with the tuning I mentioned before.
Booting up the 8.4 release ISO (now tested both i386 and amd64), the VM idles at a 13-14% CPU. This is a minimal test VM with 512MB RAM, 1 CPU core (type host), one CD drive for the ISO image, no hard drives and no NICs.
pveversion: proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
Physical hardware platform: AMD Opteron on Supermicro motherboard
pfSense: 2.1.5-RELEASE (i386) FreeBSD 8.3-RELEASE-p16
Hardware: 1 5GB disk virtio. 4 nics E1000. 512MB/1.00GB memory using baloon driver. 2 sockets (qemu32)
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pfSense: 2.1.5-RELEASE (i386) FreeBSD 8.3-RELEASE-p16
Hardware: 1 5GB disk virtio. 4 nics E1000. 512MB/1.00GB memory using baloon driver. 2 sockets (qemu32)
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
Here's something interesting. The increased CPU usage can be seen inside the VM as interrupt and system time. It's not purely on the host side. People with this problem might be able to see it reflected in the System tab of pfSense's RRD graphs page. You'll notice the blue and yellow shaded portions are noticeably higher than before the upgrade.
By now I've upgraded several more systems.
On a DL380g5 host with Xeon 5160 CPUs, I have a pfSense system configured with 1 socket, 2 cores, CPU type host, 512MB/2GB RAM, a SCSI disk and 4 virtio NICs. The firewall system is not loaded and and only has about 0.5 mbits/sec of traffic going through it. It runs OpenBGP, but this is not very active, announcing only a single prefix and not really taking any inbound routes.
As seen on the VM side: Before the upgrade, total CPU time was around 5% (basically all blue, system time), and interrupt time was not visible on the graph. After the upgrade, system time is around 10% and interrupt time is around 5%, so the total hovers around 15% with spikes to 20+%.
On the host side, this system used to be at 15% of 2 CPUs, but is now at 35-40%.
On newer physical hosts (like a Xeon X5570 server), the CPU time may be hard to see because the RRD graph is basically at 0. But the increase in system and interrupt time is still visible as thicker blue and yellow coloration at the bottom of the graph.
By now I've upgraded several more systems.
On a DL380g5 host with Xeon 5160 CPUs, I have a pfSense system configured with 1 socket, 2 cores, CPU type host, 512MB/2GB RAM, a SCSI disk and 4 virtio NICs. The firewall system is not loaded and and only has about 0.5 mbits/sec of traffic going through it. It runs OpenBGP, but this is not very active, announcing only a single prefix and not really taking any inbound routes.
As seen on the VM side: Before the upgrade, total CPU time was around 5% (basically all blue, system time), and interrupt time was not visible on the graph. After the upgrade, system time is around 10% and interrupt time is around 5%, so the total hovers around 15% with spikes to 20+%.
On the host side, this system used to be at 15% of 2 CPUs, but is now at 35-40%.
On newer physical hosts (like a Xeon X5570 server), the CPU time may be hard to see because the RRD graph is basically at 0. But the increase in system and interrupt time is still visible as thicker blue and yellow coloration at the bottom of the graph.
Last edited:
I have been able to make virtio-net work
The trick was to Disable hardware checksum offload found under System->Advanced->Networking.
These two is also disabled (I can't remember if this is the default setting)
- Hardware TCP Segmentation Offloading
- Hardware Large Receive Offloading
The trick was to Disable hardware checksum offload found under System->Advanced->Networking.
These two is also disabled (I can't remember if this is the default setting)
- Hardware TCP Segmentation Offloading
- Hardware Large Receive Offloading
I have been able to make virtio-net work
The trick was to Disable hardware checksum offload found under System->Advanced->Networking.
These two is also disabled (I can't remember if this is the default setting)
- Hardware TCP Segmentation Offloading
- Hardware Large Receive Offloading
Sorry, didn't know you had a problem with virtio NICs, or I would have mentioned the fix. (Some people prefer not to use virtio for various reasons, like I went with a scsi drive on that VM mentioned above.) That's a known issue. I believe the other two were turned off by default recently. I remember being surprised to see them set on a new install when I went in specifically to turn off the first option for use with virtio.
But that doesn't seem to be relevant to the high CPU utilization...
I am, however, starting to wonder if virtio exacerbates the high CPU load and increases interrupt time instead of decrease it. Haven't had a chance to run more in depth testing. That install I mentioned earlier is running at almost double the normal CPU and has me worried. It's one of my older systems so higher utilization tends to show up more dramatically. But really shouldn't run at 40% CPU just to pass half a megabit of traffic. ;^(
I have the same issue with the last 3.3 update and now our pfsense fw is allways with 50% CPU and with the params at the loader.conf.local now is 20% CPU usage.
pfsense 2.1.3-RELEASE (amd64)
FreeBSD 8.3-RELEASE-p16
we are also using the 3.10 kernel because the Ivy Bridge/Haswell CPU issue,
pfsense 2.1.3-RELEASE (amd64)
FreeBSD 8.3-RELEASE-p16
we are also using the 3.10 kernel because the Ivy Bridge/Haswell CPU issue,
FYI, I have some E3-1270 v3 systems running Proxmox 3.3 with the default 2.6.32-33-pve kernel. Not positive, but I think that 64bit FBSD on Haswell problem might have been resolved in the new QEMU package, in case you're not totally comfortable running non-default kernels.
A pfSense VM on a host with this CPU can idle at 2-3% typically (with selected loader tweaks above), with bursty utilization when actively passing traffic.
A pfSense VM on an HP DL360 G5 with Xeon 5160 CPUs will idle at 25-30% CPU even with the tweaks.
Our servers are HP DL360 G8 with two E5-2660 v2 and the 64bits FBSD pfsense hangs on startup with default kernel.
Until yesterday I was comfortable with the 3.10, but now with this CPU Load, not so much
you can see the increase on CPU load on the virtual pfsense under proxmox
For default kernel you must use qemu64 as processor.
As mentioned above adding this to /boot/loader.conf.local
hint.apic.0.clock=0
kern.hz=100
Reduces load considerably when using virtio.
As mentioned above adding this to /boot/loader.conf.local
hint.apic.0.clock=0
kern.hz=100
Reduces load considerably when using virtio.
Yes mir, i have add this params to loader.conf.local but still the load goes from 20% to 55%.
I think that this params are more focused for idle CPU functions, but my pfsense is very active, fw, haproxy and snort
And like say nixcz the interrupts inside pfsense now are really visible just after the upgrade, this is a very bad notice and unacceptable regresion
Just to confirm seeing this in Proxmox 3.4 with pfsense 2.1.5. Also confirmed adding kern.hz=100 to /boot/loader.conf.local with reboot and turning "Use Tablet for Pointer" off reduced idle CPU usage from approx 16% to %4 (graph attached).
Two articles I found interesting on this :
https://blog.pfsense.org/?p=115
http://serverfault.com/questions/42...ance-in-freebsd-if-dynamic-tick-mode-is-enabl
Looks like pfsense 2.2 might sort this being based on freebsd 10.1
---------------------------------------
VM Spec :
---------------------------------------
CPU : 4 x CPU Cores (kvm64) (based on physical Xeon E5640 @ 2.67Ghz - duel cpu)
Memory : 4GB
Network Device : e1000 x 7
---------------------------------------
---------------------------------------
Software :
---------------------------------------
Proxmox Version : 3.4
Pfsense : 2.1.5
Two articles I found interesting on this :
https://blog.pfsense.org/?p=115
http://serverfault.com/questions/42...ance-in-freebsd-if-dynamic-tick-mode-is-enabl
Looks like pfsense 2.2 might sort this being based on freebsd 10.1
---------------------------------------
VM Spec :
---------------------------------------
CPU : 4 x CPU Cores (kvm64) (based on physical Xeon E5640 @ 2.67Ghz - duel cpu)
Memory : 4GB
Network Device : e1000 x 7
---------------------------------------
---------------------------------------
Software :
---------------------------------------
Proxmox Version : 3.4
Pfsense : 2.1.5