Problem deleting lxc templates Proxmox VE Ceph

[Johan Blaauw]

Hi,

I have created a template of a lxc container and created subsequently a container of this template.
After changing some settings in the container I wanted to create a new template of this container and delete the first template after creating the new template. I tried to delete the first template and got the response that there are still linked images to the template. So the simplest thing to do was to delete the second template first and then delete the first template.

When I try to delete the first template I get the following error:

TASK ERROR: error with cfs lock 'storage-CT_Storage': base volume 'base-104-disk-1' is still in use by linked clones

When I try to delete the second template I get the following error:

TASK ERROR: error with cfs lock 'storage-CT_Storage': rbd error: error setting snapshot context: (2) No such file or directory

I've googled several possible solutions but none worked and resulted in the error mentioned above.

 

[Alwin]

Do a full clone of the template, not a linked one (default).

 

[Johan Blaauw]

Hi Alwin,
Thanks for your reply.

There are no clones of the templates there are only two templates.
How can I delete the two templates?

 

[Alwin]

TASK ERROR: error with cfs lock 'storage-CT_Storage': base volume 'base-104-disk-1' is still in use by linked clones

Once there is no linked CT with the template, a removal should be possible (more button).

TASK ERROR: error with cfs lock 'storage-CT_Storage': rbd error: error setting snapshot context: (2) No such file or directory

Does the rbd image to that template still exist?

When the images to the templates don't exist and a removal through the GUI/CLI is not possible, then you can remove the vmid.conf under /etc/pve/lxc/.

 

[Eirik Rye]

Once there is no linked CT with the template, a removal should be possible (more button).


Does the rbd image to that template still exist?

When the images to the templates don't exist and a removal through the GUI/CLI is not possible, then you can remove the vmid.conf under /etc/pve/lxc/.

We are having the same issue when trying to delete container templates:

On first attempt to delete (after all linked clones are removed):

2019-01-10 15:48:27.548015 7f56b1375100 -1 did not load config file, using default settings.
2019-01-10 15:48:27.653172 7f770032c100 -1 did not load config file, using default settings.
2019-01-10 15:48:27.756429 7f138504a100 -1 did not load config file, using default settings.
Removing all snapshots: 100% complete...
Removing all snapshots: 100% complete...done.
2019-01-10 15:48:28.078412 7f741fbdb100 -1 did not load config file, using default settings.
image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
TASK ERROR: error with cfs lock 'storage-ceph_storage': rbd rm 'base-128-disk-2' error: rbd: error: image still has watchers

When checking RBD in the Ceph cluster, the PVE node is the only listed watcher. Does it not recognize itself?

Then, on subsequent attempts to delete the template:

TASK ERROR: error with cfs lock 'storage-ceph_storage': rbd error: error setting snapshot context: (2) No such file or directory

The RBD image still exists in Ceph, and the Proxmox node where the template was originally created is listed as the only watcher:

root@x:~# rbd status vm/base-117-disk-1
Watchers:
        watcher=[pve-node]:0/3255163978 client.8081844 cookie=x

How can I have the Proxmox node cleanly unmount/unwatch the RBD image? I don't want to remove the files in "/etc/pve/local/lxc" until the actual backing RBD images are deleted.

 

[Alwin]

How can I have the Proxmox node cleanly unmount/unwatch the RBD image? I don't want to remove the files in "/etc/pve/local/lxc" until the actual backing RBD images are deleted.

Check on the node that is watching if it is mounted or still mapped. If it doesn't help, as a last resort, reboot the node.

 

[Eirik Rye]

I believe this may be inching close to the core issue.

VM 117 was an LXC container that was converted to a template through the Proxmox GUI. This causes the disk image to be renamed (or cloned, perhaps?)

The container template lists the following rootfs in its configuration:

root@p:~# pvesh get nodes/p/lxc/117/config | grep 117
200 OK
   "rootfs" : "ceph_storage:base-117-disk-1,size=8G",

However, the Proxmox node has the disk mapped by another name:

root@p:~# rbd showmapped | grep 117
5  vm   vm-117-disk-1   -    /dev/rbd5

In Ceph, "vm-117-disk-1" is not a valid name. It is actually named "base-117-disk-1" after the conversion to a template:

root@c:~# rbd list vm | grep 117
base-117-disk-1

My suspicion is that, when converting to a template, Proxmox renames/clones the RBD image on the Ceph storage, but somehow fails to update the local RBD mappings? Then, when attempting to delete the template, it then attempts to unmap "base-117-disk-1", which does not exist as a mapping locally?

Unmapping "vm-117-disk-1" on the Proxmox node, then removing the RBD image in Ceph seems to work:

root@p:~# rbd unmap vm/vm-117-disk-1

root@c:~# rbd rm vm/base-117-disk-1
Removing image: 100% complete...done.

Is it now safe to simply delete "/etc/pve/lxc/117.conf"?

 

[Alwin]

VM 117 was an LXC container that was converted to a template through the Proxmox GUI. This causes the disk image to be renamed (or cloned, perhaps?)

They are renamed when converted into a template.

My suspicion is that, when converting to a template, Proxmox renames/clones the RBD image on the Ceph storage, but somehow fails to update the local RBD mappings? Then, when attempting to delete the template, it then attempts to unmap "base-117-disk-1", which does not exist as a mapping locally?

I will try to reproduce this on my system.

Is it now safe to simply delete "/etc/pve/lxc/117.conf"?

If there are no linked clones referencing the template then yes. And if there are then they miss their base image.

 

[Eirik Rye]

Any update on this? Is this issue resolved in later releases?

 

[da-alb]

It seems not

Any update on this? Is this issue resolved in later releases?

It doesn't seem so. I have converted an lxc vm to template and now I cannot remove the template. I use Ceph as storage.

 

[Alwin Antreich]

converted an lxc vm to template

Which one, container or VM? The template can only be deleted if there is no VM or CT linked to it.

 

[da-alb]

Which one, container or VM? The template can only be deleted if there is no VM or CT linked to it.

LXC container converted to LXC template and it's not linked to anything...