Images of snapshots don't always get deleted

[Ingo S]

Hi forum

Today I was doing some cleanup on our Ceph storage and i found that some images of snapshots of some machines are still there while the corresponding snapshot has been deleted long ago.
An example is our machine 401:
root@vm-1:~# rbd ls -l HDD_Storage [...] vm-312-disk-1 60 GiB 2 excl vm-401-state-pre_update 32 GiB 2 excl vm-501-disk-1 500 GiB 2 excl
There is an image of a machine state still there, but the machine itself has no snapshots


There are some other machines that have state images and other snapshot images of rbd devices that do not have a parent snapshot.

This might be a bug but i don't know how to replicate it. If i do a snapshot and delete it, every image gets deleted as expected.
Any advice on this for further investigation?
I guess its safe to delete the state image?

 

[fabian]

yeah those should be safe to remove if the snapshot itself does not exist anymore in the config. if you can reproduce it with current PVE packages installed please file a bug report and include details about your setup and the steps you take to reproduce - otherwise it's quite likely that it was an issue in the past.

 

[init7]

After Ceph suddenly became full, even though only a few VMs were added, I have now noticed that I have the same thing.

• Most snapshots and state are remove correctly.
• Small amount of snapshots stay on ceph, even deleted via the web gui.
• More, but still a few states (RAM dump) stay on ceph, also removed via web gui.
• Only difference to the post from Ingo S, my old snapshots and states don't have the excl.
• I can remove them with the rdb command and ceph usage goes down.
• One snapshot could not be deleted via web gui since the disk configuration changed since then. The other left over snapshots are before and after this event. I don't think there is a connection with this.
• I'm on version 6.4-13, haven't had time to update to 7 yet.

I checked a few of the cases and the log entry in the task list indicate no error when deleting the snapshots. I will now observe this a bit in the coming weeks. If it should happen again, which logs should I pay attention to? Under /var/log/pve I don't see any errors for the not removed snapshots.

 

[fabian]

the task logs most likely.