Server crash until high I/O on KVM PVE3.4/4

[mcbarlo]

Server is crashing after 5-15 minutes when VM use much I/O for example on quota check. Logs are empty.

Server config:

- i3-4130T
- 4 GB RAM
- 2x SSD Intel 320 40GB for OS and 4x WD Green for storage (two ZFS pools)

Both storages are Directory type.

[COLOR=#000000][FONT=monospace]proxmox-ve: 4.0-16 (running kernel: 4.2.2-1-pve)
pve-manager: 4.0-48 (running version: 4.0-48/0d8559d0)
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve1
corosync-pve: 2.3.5-1libqb0: 0.17.2-1
pve-cluster: 4.0-22
qemu-server: 4.0-30
pve-firmware: 1.1-7
libpve-common-perl: 4.0-29
libpve-access-control: 4.0-9
libpve-storage-perl: 4.0-25
pve-libspice-server1: 0.12.5-1
vncterm: 1.2-1
pve-qemu-kvm: 2.4-9
pve-container: 1.0-6
pve-firewall: 2.0-12
pve-ha-manager: 1.0-9
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.3-1
lxcfs: 0.9-pve2
cgmanager: 0.37-pve2
criu: 1.6.0-1
zfsutils: 0.6.5-pve4~jessie[/FONT][/COLOR]

Here is VM config:

boot: cdnbootdisk: virtio0
cores: 3
ide2: none,media=cdrom
memory: 2048
name: backup
net0: virtio=F2:2E:2D:F4:57:60,bridge=vmbr0
net1: virtio=2E:B0:FE:5B:0C:FA,bridge=vmbr1
numa: 0
ostype: l26
smbios1: uuid=f8c710f8-7b9a-4481-a700-4dcaa2973b78
sockets: 1
virtio0: local:108/vm-108-disk-1.qcow2,cache=writethrough,discard=on,size=10G
virtio1: backup:108/vm-108-disk-1.raw,cache=writethrough,size=5300G

I tried this solutions without success:

- change disk type to ide,
- disable ACPI and APIC in host kernel,
- disable ACPI in guest,
- disable C-state in BIOS,
- change qemu machine type.

Issue exists on PVE 3.4 and 4.0 with kernel 2.6, 3.10 and newest 4.2. I ran out of ideas.

 

[sigxcpu]

First thing to do would be a memory test.

 

[mcbarlo]

Memtest passed without errors. I generated high I/O on host without any problems. I guess hardware is ok.

 

[morph027]

ZFS really eats and needs memory. You should at least use 8GB RAM and limit ZFS usage to 4 GB. But even that is kind of low end for what you need to have fun with ZFS. I'd recommend at least 16 to have enough for ZFS and VMs. You might limit ZFS to 2 GB or so now, but then it get's incredibly slow, also you only have around 1GB left for the VM. Or just stick with LVM/mdadm instead of ZFS if you can't upgrade memory.

 

[mcbarlo]

If this is only RAM amount issue I can upgrade. RAM is cheap. This is server for backup and our third DNS. I don't need performance but energy save.

I will try limit ZFS memory. Hope this help.

 

[mcbarlo]

I added 4GB RAM and VMs running 8 days without crash.

 

[fireon]

ZFS really eats and needs memory. You should at least use 8GB RAM and limit ZFS usage to 4 GB. But even that is kind of low end for what you need to have fun with ZFS. I'd recommend at least 16 to have enough for ZFS and VMs. You might limit ZFS to 2 GB or so now, but then it get's incredibly slow, also you only have around 1GB left for the VM. Or just stick with LVM/mdadm instead of ZFS if you can't upgrade memory.

Can you tell me please why i need a lot of RAM for ZFS? What do ZFS with this RAM. Is it a Cash? What is when i have no UPS and the machine crashes?

Thank you
Best Regards