Crash with reboot

[VoipTel]

Hello fellas

Our environment is composed of 1 Server (HPE Microserver Gen10) with 16 GB Ram and a local ZFS storage 2TB disks in md-raid 1 (same disks for proxmox installation and vm storage).

Standard Proxmox installation version 5.0-30.

Over this installation we run the following VM:

- 1 linux VM (Zentyal) with kernel 4.4.0-93, 2 GB Ram, 100GB HDD

- 1 linux VM (Ubuntu 16.04 LTS) with kernel 4.4.0-93, 2GB Ram, 200GB HDD

- 1 Windows 2016 VM with 6 GB Ram, 100GB HDD

We set up a cron to backup data from Zentyal to Ubuntu using duplicity.

When duplicity backup starts, I can see intense HDD IO on the Zentyal server and then after a couple of minutes the entire Proxmox server suddenly reboot.

We checked the log files but they seem to be truncated and we cant' see anything abnormal.

Versions of installed software are:

proxmox-ve: 5.0-20 (running kernel: 4.10.17-2-pve)
pve-manager: 5.0-30 (running version: 5.0-30/5ab26bc)
pve-kernel-4.10.17-2-pve: 4.10.17-20
pve-kernel-4.10.15-1-pve: 4.10.15-15
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-14
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.0-4
pve-container: 2.0-15
pve-firewall: 3.0-2
pve-ha-manager: 2.0-2
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.11-pve17~bpo90

Any suggestion?

 

[GadgetPig]

I'm not the expert, but I recall using ZFS requires adding more memory. Maybe ZFS used more ram during file copies than it free'd up, causing the crash.

see here:

https://forum.proxmox.com/threads/zfs-server-crash.32675/
https://forum.proxmox.com/threads/zfs-uses-ram-equal-to-files-being-written-to-it.28837/

From the links above, the solution was given by Fabian: "you can either put more memory into the box (recommended), limit the VM memory and/or limit the ARC"

In your case I assume your 2 memory slots are already used; You could try dropping zentyal+ubuntu to use 1gb and windows to 4gb and try from there. But long term, I think 32GB memory is the best option.

 

[fireon]

16GB Memory and this VM's are very very low for ZFS. Minimal 32GB, really Minimal. And disable swap on ZFS. If you would like to have swap, enable it on a non ZFS-Partition.
You can limit the ARC as @GadgetPig discribes. https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage