Snapshots kill the VM, sometimes

[grefabu]

Hi,

I've a randomly problem with snapshots. Randomly when take a snapshot the VM is killed and I've following message:

VM 103 not running
snapshot create failed: starting cleanup
2021-07-17 09:55:40.357 7f9708ff9700 -1 librbd::image::PreRemoveRequest: 0x564b565b66b0 handle_exclusive_lock: cannot obtain exclusive lock - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
error during cfs-locked 'storage-cephpool003' operation: rbd rm 'vm-103-state-Update' error: rbd: error: image still has watchers
TASK ERROR: VM 103 not running

It occoure thet an snapshot was shown on the RDB list. Over GUI I can't delete the wrong snapshot, with rbd rm I could remove it and the System was clean again.
But today I test it, and the snapshot was without problem taken,...

Any idea?

Gregor

 

[mira]

Please provide the output of `pveversion -v`.

 

[grefabu]

proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-1-pve: 5.3.18-1
ceph: 14.2.16-pve1
ceph-fuse: 14.2.16-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

 

[grefabu]

I know I've to update to a actual version, I plan it in the next time,...

 

[mira]

This could be the same issue that is already fixed in newer versions. Please consider upgrading
Do you have `QEMU` entries in the journal around that time?

Do you live migrate the VMs sometimes before taking snapshots? We could trigger such an issue reliably by running a PBS backup, then live migrating the VM to a different node and taking a snapshot afterwards.