[SOLVED] CT is locked (snapshot-delete)

[gazbill]

Hello,

Ran some scheduled backups last night and this morning all containers and VM's were showing grey questions marks, which later managed to resolve by restarting a service in pve shell. Also a lock appeared on one of my containers and it was stopped. Tried to start it but failed with the subject error and I'm unable to remove the snapshot.

root@pve:~# pct unlock 101
root@pve:~# pct delsnapshot 101 vzdump
lvremove snapshot 'pve/snap_vm-101-disk-0_vzdump' error:   Failed to find logical volume "pve/snap_vm-101-disk-0_vzdump"
root@pve:~# vgscan
  Found volume group "pve" using metadata type lvm2
root@pve:~# lvs
  LV                       VG  Attr       LSize   Pool Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  data                     pve twi-aotz-- 338.23g                    13.83  0.73                           
  root                     pve -wi-ao----  96.00g                                                           
  snap_vm-104-disk-0_clean pve Vri---tz-k  32.00g data vm-104-disk-0                                       
  swap                     pve -wi-ao----   7.62g                                                           
  vm-100-disk-0            pve Vwi-aotz--   4.00g data               80.74                                 
  vm-101-disk-0            pve Vwi-aotz--  64.00g data               33.82                                 
  vm-102-disk-0            pve Vwi-aotz--   4.00g data               55.89                                 
  vm-103-disk-0            pve Vwi-aotz--   4.00m data               0.00                                   
  vm-103-disk-1            pve Vwi-aotz--  32.00g data               25.10                                 
  vm-104-disk-0            pve Vwi-a-tz--  32.00g data               29.09                                 
  vm-104-state-clean       pve Vwi-a-tz--  <4.49g data               36.81

How can I resolve this, and why the issues seemingly after the backup?

I only setup Proxmox over the weekend and it was all working perfectly until today.

 

[fiona]

Hi,
you can force delete the snapshot by specifying --force. Alternatively, the vzdump snapshot will automatically be cleaned up before the next backup.

As to why the issue happened in the first place, please share the full backup task log as well as the system journal from around that time.

 

[gazbill]

Brilliant,

pct delsnapshot 101 vzdump --force

fixed it. It's odd that

lvremove -f /dev/pve/snap_vm-101-disk-0_vzdump

didn't work as the force flag was set (not included in my original description), but presume it was labelled incorrectly or something.

Regarding the logs, I'll see if I can get those uploaded later, many thanks.

 

[gazbill]

Backup ran again last night at 3am and it's still running (+5 hours), stuck at "create storage snapshot 'vzdump'" on my docker container 101. The container is running fuse.

INFO: starting new backup job: vzdump 100 101 103 --prune-backups 'keep-daily=7' --mailto billettg@gmail.com --quiet 1 --fleecing 0 --mailnotification always --compress zstd --notes-template '{{guestname}}' --storage local --mode snapshot
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2025-02-05 03:00:03
INFO: status = running
INFO: CT Name: caddy
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
Logical volume "snap_vm-100-disk-0_vzdump" created.
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-100-2025_02_05-03_00_03.tar.zst'
INFO: Total bytes written: 2711429120 (2.6GiB, 93MiB/s)
INFO: archive file size: 836MB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-daily=7
INFO: pruned 0 backup(s)
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-100-disk-0_vzdump" successfully removed.
INFO: Finished Backup of VM 100 (00:00:29)
INFO: Backup finished at 2025-02-05 03:00:32
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2025-02-05 03:00:32
INFO: status = running
INFO: CT Name: docker
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'

pve-manager/8.3.0/c1689ccb1065a83b (running kernel: 6.8.12-4-pve)

pct list - this command just hangs

How to remediate/prevent this happening?

 

[gazbill]

Server was unresponsive. Luckily I put a smart plug on it and managed to hard reset. Come back up and accessible but back to the same locked container situation again. Clearly there is something preventing the backup from completing on this LXC. Fuse related?

 

[fiona]

Yes, you cannot use a container with fuse and snapshot mode backup, see: https://pve.proxmox.com/pve-docs/chapter-pct.html#_fuse_mounts

 

[gazbill]

Thanks Fiona, I've removed 101 from the backup job, and put it on it's own and using "stop" mode. Appears to be working fine now.

Many thanks!