I shut down a VM (Windows 11) and I wanted to take a snapshot. I realized the VM was not yet shut down because it was taking a snapshot of the RAM so I cancelled it while it was snapshoting.
Unfortunately, this caused the VM to get stuck in a locked state, which I cannot get it out of.
Original error when I tried to delete snapshot:
And now I get:
Task error when deleting snapshot:
I tried:
qm unlock vmid
and:
sudo systemctl stop pve-cluster
sudo rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
sudo systemctl start pve-cluster
on every node.
PVE Version:
Any ideas on what I can do to clean up this mess?
Unfortunately, this caused the VM to get stuck in a locked state, which I cannot get it out of.
Original error when I tried to delete snapshot:
Code:
rying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
trying to acquire cfs lock 'storage-ceph_hdd' ...
TASK ERROR: cfs-lock 'storage-ceph_hdd' error: got lock request timeoutAnd now I get:
Code:
()
trying to acquire cfs lock 'storage-ceph_hdd' ...
Removing image: 1% complete...
Removing image: 2% complete...
Removing image: 3% complete...
Removing image: 4% complete...
Removing image: 5% complete...
Removing image: 6% complete...
Removing image: 7% complete...
Removing image: 8% complete...
Removing image: 9% complete...
Removing image: 10% complete...
Removing image: 11% complete...
Removing image: 12% complete...
Removing image: 13% complete...
Removing image: 14% complete...
Removing image: 15% complete...
Removing image: 16% complete...
Removing image: 17% complete...
Removing image: 18% complete...
Removing image: 19% complete...
Removing image: 20% complete...
Removing image: 21% complete...
Removing image: 22% complete...
Removing image: 23% complete...
Removing image: 24% complete...
Removing image: 25% complete...
Removing image: 26% complete...
Removing image: 27% complete...
Removing image: 28% complete...
Removing image: 29% complete...
Removing image: 30% complete...
Removing image: 31% complete...
Removing image: 32% complete...
Removing image: 33% complete...
Removing image: 34% complete...
Removing image: 35% complete...
Removing image: 36% complete...
Removing image: 37% complete...
Removing image: 38% complete...
Removing image: 39% complete...
Removing image: 40% complete...
Removing image: 41% complete...
Removing image: 42% complete...
Removing image: 43% complete...
Removing image: 44% complete...
Removing image: 45% complete...
Removing image: 46% complete...
Removing image: 47% complete...
Removing image: 48% complete...
Removing image: 49% complete...
Removing image: 50% complete...
Removing image: 51% complete...
Removing image: 52% complete...
Removing image: 53% complete...
Removing image: 54% complete...
Removing image: 55% complete...
Removing image: 56% complete...
Removing image: 57% complete...
Removing image: 58% complete...
Removing image: 59% complete...
Removing image: 60% complete...
Removing image: 61% complete...
Removing image: 62% complete...
Removing image: 63% complete...
Removing image: 64% complete...
Removing image: 65% complete...
Removing image: 66% complete...
Removing image: 67% complete...
Removing image: 68% complete...
Removing image: 69% complete...
Removing image: 70% complete...
Removing image: 71% complete...
Removing image: 72% complete...
Removing image: 73% complete...
Removing image: 74% complete...
Removing image: 75% complete...
Removing image: 76% complete...
Removing image: 77% complete...
Removing image: 78% complete...
Removing image: 79% complete...
Removing image: 80% complete...
Removing image: 81% complete...
Removing image: 82% complete...
Removing image: 83% complete...
Removing image: 84% complete...
Removing image: 85% complete...
Removing image: 86% complete...
Removing image: 87% complete...
Removing image: 88% complete...
Removing image: 89% complete...
Removing image: 90% complete...
Removing image: 91% complete...
Removing image: 92% complete...
Removing image: 93% complete...
Removing image: 94% complete...
Removing image: 95% complete...
Removing image: 96% complete...
Removing image: 97% complete...
Removing image: 98% complete...
Removing image: 99% complete...
Removing image: 100% complete...done.
TASK ERROR: rbd error: error setting snapshot context: (2) No such file or directoryTask error when deleting snapshot:
Code:
TASK ERROR: rbd error: error setting snapshot context: (2) No such file or directoryI tried:
qm unlock vmid
and:
sudo systemctl stop pve-cluster
sudo rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
sudo systemctl start pve-cluster
on every node.
PVE Version:
Code:
Header
Proxmox
Virtual Environment 7.1-8
Search
Node 'VMHost4'
Hour (average)
CPU usage 8.25% of 24 CPU(s)
IO delay 0.03%
Load average 3.00,3.83,4.86
RAM usage 46.31% (58.30 GiB of 125.89 GiB)
KSM sharing 0 B
/ HD space 8.30% (7.84 GiB of 94.37 GiB)
SWAP usage 0.00% (0 B of 31.00 GiB)
CPU(s) 24 x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2 Sockets)
Kernel Version Linux 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100)
PVE Manager Version pve-manager/7.1-8/5b267f33
Repository Status Proxmox VE updates Non production-ready repository enabled!
Server View
Logs
()
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-8 (running version: 7.1-8/5b267f33)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
ceph: 15.2.15-pve1
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3Any ideas on what I can do to clean up this mess?