VM hangs or corrupts during nightly backups

[decibel83]

Hi,
I'm experiencing a very big issue on some virtual machines which I found hanged or with corrupted file system especially in the morning at the beginning of the working hours (corrupted file system means that I need to reboot the VM and launch fsck to fix).

These problems are happening on virtual machines which have a very big data load (data which change very frequently), so for this reason I'm thinking that this issue could be related to backups.

Backupd are made to Proxmox Backup Server, and local storage on Proxmox nodes is ZFS on SSD or NVMe drives.

Issues have a similar behavior: I saved some screenshots of console when VMs are hanged or corrupted:









This is the pveversion -v output:

root@node3:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph: 14.2.16-pve1
ceph-fuse: 14.2.16-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Could you help me to find out what's going on, please?

Thank you very much!
Bye

 

[spirit]

do you have tried to upgrade to last proxmox 6.4 ?
(mainly for pve-qemu package).
I'm seeing
https://git.proxmox.com/?p=pve-qemu.git;a=shortlog

 

[decibel83]

Thanks!
Should I restart every VM (I will reboot hosts later)?

Thank you!

 

[spirit]

Thanks!
Should I restart every VM (I will reboot hosts later)?

Thank you!

yes, vm need to be restarted (stop/start) after qemu upgrade.