[SOLVED] High CPU Load with KVM and ZFS

[Markus Vetter]

Hello,

I have a big problem with high CPU load under the following conditions.

After a few days, so approximately after 3-4 days, a second KVM machine (no matter which) receives very high CPU load. With top i see that "z_wr_iss" is busy completely. A reboot fixes the problem and in the first few days everything runs smoothly. If the vm is in idle state, the load remains low. But even an apt-get immediately jumps to 100% CPU load. Interestingly, the lxc-vms are not affected. Here everything runs smoothly.

My Setup
Proxmox Installation with ZFS mirror
Disk 1: SanDisk SDSSDHII960G (SSD 960 GB)
Disk 2: Crucial_CT960M500SSD1 (SSD 960 GB)
Ram: 16 GB
CPU: Xeon E3-1225 v5
8 lxc VM
2 KVM VM (Debian,Ubuntu)

pve-manager/4.1-22/aca130cf (running kernel: 4.2.8-1-pve)

pveperf
CPU BOGOMIPS: 26487.96
REGEX/SECOND: 3046630
HD SIZE: 526.36 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 133.85
DNS EXT: 57.21 ms
DNS INT: 0.90 ms

Greetings Markus

 

[spirit]

Hi,
from my point of view, 16GB is too low for zfs + 8lxc + 2kvm

you can limit zfs memory usage :
https://pve.proxmox.com/wiki/Storage:_ZFS#Limit_ZFS_memory_usage

But I really think you don't have enough memory.

 

[Markus Vetter]

Hi,
from my point of view, 16GB is too low for zfs + 8lxc + 2kvm

you can limit zfs memory usage :
https://pve.proxmox.com/wiki/Storage:_ZFS#Limit_ZFS_memory_usage

But I really think you don't have enough memory.

The 8 lxc are very small ones. Mostly tine2.0, owncloud, logitechmediaserver, fhem etc.

free
             total       used       free     shared    buffers     cached
Mem:      16231316   15965960     265356     170484          0     573800
-/+ buffers/cache:   15392160     839156
Swap:     15728636     165460   15563176

Do you really mean that this could be the cause? The lxc´s and the kvm´s runs in idle most time. If I use only one kvm (Univention UCS) this machine behaves as expected and the host runs smoothly. If I then start an ordinary Linux Distribution like Ubuntu Desktop 15.04 the load increases til 100 %.

 

[morph027]

I got a similiar setup:

• HP Microserver
• 16 GB RAM
• Xeon E3-1220L V2
  
• Samsung SSD 840 PRO @ ZFS Mirror @ LUKS

pveperf /vm-400g/
CPU BOGOMIPS:  18357.92
REGEX/SECOND:  1686637
HD SIZE:  287.02 GB (vm-400g)
FSYNCS/SECOND:  500.81

I've limited my zfs arc cache to 8GB and it runs quite smooth. Putting heavy I/O onto a kvm (e.g. test installing and dist-upgrading a new [insert-your-favourite-linux] release creates some serious load in htop, but everything stays responsible.

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=8599934592
# do not forget to run update-initramfs -k all -u

 

[LnxBil]

I've limited my zfs arc cache to 8GB and it runs quite smooth. Putting heavy I/O onto a kvm (e.g. test installing and dist-upgrading a new [insert-your-favourite-linux] release creates some serious load in htop, but everything stays responsible.

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=8599934592
# do not forget to run update-initramfs -k all -u

This is the default on Linux, so you did not limit anything. Please refer to https://pve.proxmox.com/wiki/Storage:_ZFS#Limit_ZFS_memory_usage

 

[LnxBil]

The 8 lxc are very small ones. Mostly tine2.0, owncloud, logitechmediaserver, fhem etc.

free
             total       used       free     shared    buffers     cached
Mem:      16231316   15965960     265356     170484          0     573800
-/+ buffers/cache:   15392160     839156
Swap:     15728636     165460   15563176

Do you really mean that this could be the cause? The lxc´s and the kvm´s runs in idle most time. If I use only one kvm (Univention UCS) this machine behaves as expected and the host runs smoothly. If I then start an ordinary Linux Distribution like Ubuntu Desktop 15.04 the load increases til 100 %.

Swapping is always a bad sign. There are a lot of entries for swap on ZFS on this forum and how to tweak it. Maybe you should try that also. You can also install zram-config from ubuntu to have an automatic zram as swap.

 

[Markus Vetter]

I've now invested in new ram ( 2 x 16 GB) . Now I have 48 GB Ram and the problem is solved . Thank you for reporting.