High CPU Idle Usage with pfsense/OpenBSD

Discussion in 'Proxmox VE: Installation and configuration' started by dennisd, Sep 14, 2014.

  1. dennisd

    dennisd New Member

    Joined:
    Oct 12, 2013
    Messages:
    3
    Likes Received:
    0
    Hey all,
    since I've updated on the latest packages (nonStable/free Repo) I did notice a cpu increase/utilisation for my KVM pfsense/freebsd machine.

    Does anybody else has the same problem? Is there a workaround?

    Thanks
     
  2. nixcz

    nixcz New Member

    Joined:
    Sep 16, 2014
    Messages:
    11
    Likes Received:
    0
    Yes, I happened to upgrade right at the release of 3.3 (great work, and thanks to the devs), but ran into this problem. I use pfSense and other FreeBSD VMs heavily and was dismayed to see idle VMs hit 10-15% of two allocated CPU cores.

    For me, this only seems to affect FreeBSD version 8. Current pfSense 2.1.5 is based on FreeBSD 8.3 so it's affected. I have an 8.x VM that has this problem, and booting the 8.4 release ISO also shows it. I booted the 9.1, 9.3, 10.0, and 10.1-BETA release ISOs, and they seem to work just fine and do NOT show the problem. I only tried amd64; have no idea how i386 behaves.

    Note that disabling the tablet pointer in the VM options tab seems to reduce CPU usage by a noticeable 2-3% regardless of FreeBSD version, so that's low hanging fruit to hit first.

    After a lot of trial and error, it seems like we have to resort to some "old" tricks like setting kern.hz to 100 instead of the default 1000. I added the following in my /boot/loader.conf.local (in addition to the virtio lines) on one of my pfSense VMs and will run it for a while before upgrading other hosts to Proxmox 3.3.

    Code:
    hint.apic.0.clock=0
    kern.hz=100
    
    You can find some other options at the bottom of this well known thread. Up to you to test and see what works best.

    https://communities.vmware.com/thread/87794

    Note that changing the CPU type, number of cores, virtio vs. non-virtio devices, etc. didn't seem to have much effect for me. YMMV.

    And finally, I don't mean this as a complaint, but as a general observation, that FBSD support on Proxmox/KVM has been a lot more fragile than I would like to see for production systems. I count at least three major problems in the last ~two years that have been addressed (or not) in varying amounts of time.

    There was that SeaBIOS issue, which had a published workaround within a day, the Ivy Bridge/Haswell CPU issue, which AFAIK still requires using a non-default kernel (read a report that the 3.10 non-OpenVZ kernel fixes this?), and now this idle CPU regression.
     
  3. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,476
    Likes Received:
    95
    My PfSense runs stable with cpu usage between 3.5-4.0%.
     
  4. nixcz

    nixcz New Member

    Joined:
    Sep 16, 2014
    Messages:
    11
    Likes Received:
    0
    Hi mir, can you share some details about your setup?

    Physical hardware platform, pfSense version and i386 or amd64, any special tuning parameters, did you upgrade to Proxmox 3.3 from the no-sub repo, etc?

    I'm seeing the problem on a Nehalem class server (Supermicro motherboard). I can get idle CPU under 4% only with the tuning I mentioned before.

    Booting up the 8.4 release ISO (now tested both i386 and amd64), the VM idles at a 13-14% CPU. This is a minimal test VM with 512MB RAM, 1 CPU core (type host), one CD drive for the ISO image, no hard drives and no NICs.

    pveversion: proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
     
  5. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,476
    Likes Received:
    95
    Physical hardware platform: AMD Opteron on Supermicro motherboard
    pfSense: 2.1.5-RELEASE (i386) FreeBSD 8.3-RELEASE-p16
    Hardware: 1 5GB disk virtio. 4 nics E1000. 512MB/1.00GB memory using baloon driver. 2 sockets (qemu32)
    proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
     
  6. nixcz

    nixcz New Member

    Joined:
    Sep 16, 2014
    Messages:
    11
    Likes Received:
    0
    Thanks for the info. Since you're on Opteron, I'm not sure if we can make any meaningful comparisons. I haven't used any of those in 7-8 years. Would love to know what other people are seeing.
     
  7. nixcz

    nixcz New Member

    Joined:
    Sep 16, 2014
    Messages:
    11
    Likes Received:
    0
    Here's something interesting. The increased CPU usage can be seen inside the VM as interrupt and system time. It's not purely on the host side. People with this problem might be able to see it reflected in the System tab of pfSense's RRD graphs page. You'll notice the blue and yellow shaded portions are noticeably higher than before the upgrade.

    By now I've upgraded several more systems.

    On a DL380g5 host with Xeon 5160 CPUs, I have a pfSense system configured with 1 socket, 2 cores, CPU type host, 512MB/2GB RAM, a SCSI disk and 4 virtio NICs. The firewall system is not loaded and and only has about 0.5 mbits/sec of traffic going through it. It runs OpenBGP, but this is not very active, announcing only a single prefix and not really taking any inbound routes.

    As seen on the VM side: Before the upgrade, total CPU time was around 5% (basically all blue, system time), and interrupt time was not visible on the graph. After the upgrade, system time is around 10% and interrupt time is around 5%, so the total hovers around 15% with spikes to 20+%.

    On the host side, this system used to be at 15% of 2 CPUs, but is now at 35-40%.

    On newer physical hosts (like a Xeon X5570 server), the CPU time may be hard to see because the RRD graph is basically at 0. But the increase in system and interrupt time is still visible as thicker blue and yellow coloration at the bottom of the graph.
     
    #7 nixcz, Sep 17, 2014
    Last edited: Sep 18, 2014
  8. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,476
    Likes Received:
    95
    I have been able to make virtio-net work:)
    The trick was to Disable hardware checksum offload found under System->Advanced->Networking.

    These two is also disabled (I can't remember if this is the default setting)
    - Hardware TCP Segmentation Offloading
    - Hardware Large Receive Offloading
     
  9. nixcz

    nixcz New Member

    Joined:
    Sep 16, 2014
    Messages:
    11
    Likes Received:
    0
    Sorry, didn't know you had a problem with virtio NICs, or I would have mentioned the fix. (Some people prefer not to use virtio for various reasons, like I went with a scsi drive on that VM mentioned above.) That's a known issue. I believe the other two were turned off by default recently. I remember being surprised to see them set on a new install when I went in specifically to turn off the first option for use with virtio.

    But that doesn't seem to be relevant to the high CPU utilization...

    I am, however, starting to wonder if virtio exacerbates the high CPU load and increases interrupt time instead of decrease it. Haven't had a chance to run more in depth testing. That install I mentioned earlier is running at almost double the normal CPU and has me worried. It's one of my older systems so higher utilization tends to show up more dramatically. But really shouldn't run at 40% CPU just to pass half a megabit of traffic. ;^(
     
  10. vmalaga

    vmalaga New Member

    Joined:
    Jul 3, 2012
    Messages:
    4
    Likes Received:
    0
    I have the same issue with the last 3.3 update and now our pfsense fw is allways with 50% CPU and with the params at the loader.conf.local now is 20% CPU usage.

    pfsense 2.1.3-RELEASE (amd64)
    FreeBSD 8.3-RELEASE-p16

    we are also using the 3.10 kernel because the Ivy Bridge/Haswell CPU issue,
     
  11. nixcz

    nixcz New Member

    Joined:
    Sep 16, 2014
    Messages:
    11
    Likes Received:
    0
    FYI, I have some E3-1270 v3 systems running Proxmox 3.3 with the default 2.6.32-33-pve kernel. Not positive, but I think that 64bit FBSD on Haswell problem might have been resolved in the new QEMU package, in case you're not totally comfortable running non-default kernels.

    A pfSense VM on a host with this CPU can idle at 2-3% typically (with selected loader tweaks above), with bursty utilization when actively passing traffic.

    A pfSense VM on an HP DL360 G5 with Xeon 5160 CPUs will idle at 25-30% CPU even with the tweaks.
     
  12. vmalaga

    vmalaga New Member

    Joined:
    Jul 3, 2012
    Messages:
    4
    Likes Received:
    0
    Our servers are HP DL360 G8 with two E5-2660 v2 and the 64bits FBSD pfsense hangs on startup with default kernel.
    Until yesterday I was comfortable with the 3.10, but now with this CPU Load, not so much :mad:

    you can see the increase on CPU load on the virtual pfsense under proxmox

    pfs01_load.png
     
  13. mir

    mir Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 14, 2012
    Messages:
    3,476
    Likes Received:
    95
    For default kernel you must use qemu64 as processor.

    As mentioned above adding this to /boot/loader.conf.local
    hint.apic.0.clock=0
    kern.hz=100

    Reduces load considerably when using virtio.
     
  14. vmalaga

    vmalaga New Member

    Joined:
    Jul 3, 2012
    Messages:
    4
    Likes Received:
    0
    Yes mir, i have add this params to loader.conf.local but still the load goes from 20% to 55%.
    I think that this params are more focused for idle CPU functions, but my pfsense is very active, fw, haproxy and snort

    And like say nixcz the interrupts inside pfsense now are really visible just after the upgrade, this is a very bad notice and unacceptable regresion


    pfsense_interrupts.png
     
  15. oxygen8.ireland

    oxygen8.ireland New Member

    Joined:
    Mar 2, 2015
    Messages:
    1
    Likes Received:
    0
    Just to confirm seeing this in Proxmox 3.4 with pfsense 2.1.5. Also confirmed adding kern.hz=100 to /boot/loader.conf.local with reboot and turning "Use Tablet for Pointer" off reduced idle CPU usage from approx 16% to %4 (graph attached).

    Two articles I found interesting on this :

    https://blog.pfsense.org/?p=115
    http://serverfault.com/questions/42...ance-in-freebsd-if-dynamic-tick-mode-is-enabl

    Looks like pfsense 2.2 might sort this being based on freebsd 10.1


    ---------------------------------------
    VM Spec :
    ---------------------------------------
    CPU : 4 x CPU Cores (kvm64) (based on physical Xeon E5640 @ 2.67Ghz - duel cpu)
    Memory : 4GB
    Network Device : e1000 x 7
    ---------------------------------------


    ---------------------------------------
    Software :
    ---------------------------------------
    Proxmox Version : 3.4
    Pfsense : 2.1.5


    proxmox_cpu_usage_pfsense.png
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice