Issue with incorrect cfs lock

[Ralms]

Hi everyone,

I've been exploring Proxmox for the first time over the last 2 weeks.
Overall looks a very promising but I have to be honest, I hate the storage management with a passion, what a mess.

I want to deploy a Linux Container on top of a iSCSI disk, never used Linux Containers so don't know if this doesn't make sense, however I just want to separate storage from the host completely so I can move things around more easily.
So here is the steps I've took:

1. Select "Datacenter > Storage", do "Add > iSCSI" and created my new target.
2. As I found out that is not possible to create containers on top of raw iSCSI targets, I went ahead and created a LVM on top of the iSCSI Disk created in the previous step.
   For reference, the volume group I created was 'iSCSI-share-lvm".
3. Attempt to create the CT, failed.... because of space. I've set the block disk in FreeNAS to 10Gib and 10GB in the CT.
4. Ok, go to FreeNAS and expand the disk to 10.2Gib to realize that Proxmox didn't assume the change.
5. Delete LVM, try to recreate and here is when the fun starts.... error:
   create storage failed: error with cfs lock 'file-storage_cfg': device '/dev/disk/by-id/scsi-36589cfc000000bc44da4a8ba419be12e' is already used by volume group 'iSCSl-share-lvm'
   (500)
6. I've tried to find out how to delete this freaking volume group, can't find a way.
   So I rebooted the host, same issue.
   
7. Delete iSCSI target, re-add it with the same ID
8. Try to create the LVM again, same error.
9. I'm so upset with this stupid nonsense errors that I just simply don't care, so I went ahead and deleted:
   Delete iSCSI target in the portal.
   Run in shell: rm /dev/disk/by-id/scsi-36589cfc000000bc44da4a8ba419be12e
   Run in shell: pvs (this returned me the mount of the volume group)
   Run in shell: rm /dev/sdb (Volume group GONE! what a miracle)
   I knew this could cause errors, but I just didn't care anymore.
10. Try to re-add/create everything again and the bloody lock is still there.... Error creating the LVM:
    create storage failed: error with cfs lock 'file-storage_cfg': pvcreate 'Idev/disk/by-id/scsi-
    36589cfc000000bc44da4a8ba419be12e' error: Device /dev/disk\by- dJscsi-
    36589cfc000000bc44da4a8ba419be12e not found (or ignored by filtering).
    

So what I've learned so far is that Proxmox seems to reuse the same IDs in the backgroup when we recreated or re-add things with the same name/id in the console, completely defeating the purpose of removing/deleting something.

I'm sorry for my rant, but from someone with a high level of expertise with Windows environments, this situation is just stupid on my eyes.

Could you please help me fix and remove this stupid CFS Lock which is incorrect please?

 

[t.lamprecht]

Could you please help me fix and remove this stupid CFS Lock which is incorrect please?

There's no lock, stupid or not, this is a red herring. The messages is to be interpreted as: "error in operation under locked context foo: REAL ERROR HERE"
So lets look at the real error:

pvcreate 'Idev/disk/by-id/scsi-36589cfc000000bc44da4a8ba419be12e' error: Device /dev/disk\by- dJscsi-36589cfc000000bc44da4a8ba419be12e not found (or ignored by filtering).

it seems the disk ID is not there/mapped, can you ensure you got the correct one now, as you removed this exact one in step 9?

I'm sorry for my rant, but from someone with a high level of expertise with Windows environments, this situation is just stupid on my eyes.

iSCSI, at least with CTs, may be one of the a bit worser supported storage types by Proxmox VE, so yes I can see that this can be a bit of a nuisance. @dcsapak wanted to improve CTs on iSCSI, but ran into some road blocks, IIRC, maybe he can get you a good workaround, personally I do not touch iSCSI to often in recent times, just much more happy with SDN based storage techs (e.g., ceph cluster wide, ZFS locally) - not that that helps you here.

 

[dcsapak]

I've tried to find out how to delete this freaking volume group, can't find a way.

here the general lvm documentation is very helpful
see for example 'man lvm'
for removing a volume group you need to do a 'vgremove'

lvm on iscsi is currently the only way to get container on an iscsi, and the way it is implemented is intended for
a single large lun where your vm/ct disks are on, not a lun for every vm/ct disk

a much faster way would probably have been nfs/cifs for a quick test

 

[Ralms]

Hi all,

First of all thank you for your replies.

There's no lock, stupid or not, this is a red herring. The messages is to be interpreted as: "error in operation under locked context foo: REAL ERROR HERE"
So lets look at the real error:

it seems the disk ID is not there/mapped, can you ensure you got the correct one now, as you removed this exact one in step 9?

iSCSI, at least with CTs, may be one of the a bit worser supported storage types by Proxmox VE, so yes I can see that this can be a bit of a nuisance. @dcsapak wanted to improve CTs on iSCSI, but ran into some road blocks, IIRC, maybe he can get you a good workaround, personally I do not touch iSCSI to often in recent times, just much more happy with SDN based storage techs (e.g., ceph cluster wide, ZFS locally) - not that that helps you here.

That error was after I started to recklessly delete stuff in attempt to get it to work as I don't have anything fully deployed yet.
The first one I got was in step 5:
create storage failed: error with cfs lock 'file-storage_cfg': device '/dev/disk/by-id/scsi-36589cfc000000bc44da4a8ba419be12e' is already used by volume group 'iSCSl-share-lvm'
(500)

So on step 9 I deleted that device with the shell after removing the iSCSI target in the console.

Although this device was not being used, it seems that deleting the LVM and the iSCSI target didn't remove the reference and the volume group.

here the general lvm documentation is very helpful
see for example 'man lvm'
for removing a volume group you need to do a 'vgremove'

lvm on iscsi is currently the only way to get container on an iscsi, and the way it is implemented is intended for
a single large lun where your vm/ct disks are on, not a lun for every vm/ct disk

a much faster way would probably have been nfs/cifs for a quick test

I did check the documentation, while searching I found this page:
https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM)
however doesn't have any information about that command unfortunately.

As my issue still remains being around the error "error with cfs lock 'file-storage_cfg'", where can I look to attempt to fix it please?

Thank you for your assistance,
Ralms.

 

[dcsapak]

So on step 9 I deleted that device with the shell after removing the iSCSI target in the console.

Although this device was not being used, it seems that deleting the LVM and the iSCSI target didn't remove the reference and the volume group.

you deleted /dev/sdb which was the device node not the device itself
and you deleted /dev/disk/... which is only a symlink to the device node

to get those back, reboot the server (do not simply delete some files from /dev, this only leads to headaches)

I did check the documentation, while searching I found this page:
https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM)
however doesn't have any information about that command unfortunately.

you are right, this command is not mentioned, but lvm is not a proxmox specific software but one used in linux in general,
so general linux/lvm knowledege is also applicable here

As my issue still remains being around the error "error with cfs lock 'file-storage_cfg'", where can I look to attempt to fix it please?

as thomas said, the 'cfs lock' part of the error is not the error itself but the part of the code where it occurs, namely the error occured while the api call held the lock (if the error would be related to the lock, it would say something like:
'could not acquire lock' or '... is locked')

to get into a good state, i would suggest to restart your pve host, and then remove the volume group with 'vgremove'. please see the manpage with 'man vgremove' on how to do that

 

[Ralms]

you deleted /dev/sdb which was the device node not the device itself
and you deleted /dev/disk/... which is only a symlink to the device node

to get those back, reboot the server (do not simply delete some files from /dev, this only leads to headaches)

you are right, this command is not mentioned, but lvm is not a proxmox specific software but one used in linux in general,
so general linux/lvm knowledege is also applicable here

as thomas said, the 'cfs lock' part of the error is not the error itself but the part of the code where it occurs, namely the error occured while the api call held the lock (if the error would be related to the lock, it would say something like:
'could not acquire lock' or '... is locked')

to get into a good state, i would suggest to restart your pve host, and then remove the volume group with 'vgremove'. please see the manpage with 'man vgremove' on how to do that

Hi there again,

Would like to say that I've followed your instructions and they worked perfectly.
Rebooted the host, the volume group showed up again, after that ran vgremove successfully and was able to create a new LVM from that iSCSI target.

I'm very sorry for my rant yesterday, was hot headed for sure.
Just one suggestion, add a "Remove: Volume Group" option in the portal at "Disks > LVM"

Now I'm I've been considering what you mentioned on using NFS or SMB instead although still not sure.
I'm kinda exploring how it would work if I had to move this CT to another host or recover from some disaster.

Again, thank you for your help dcsapak and t.lamprecht and my apologies for my behavior.

 

[dcsapak]

Just one suggestion, add a "Remove: Volume Group" option in the portal at "Disks > LVM"

removing data is always risky and removing an entire storage(pool) is more dangerous even so we try to avoid this having only a click away
since this is not really an operation that one needs to do very often, my personal opinion is that going to the cmdline for this is not that much of a hurdle

I'm kinda exploring how it would work if I had to move this CT to another host or recover from some disaster.

this depends if you have a cluster and shared storage. in that case you should look at HA https://pve.proxmox.com/wiki/High_Availability (be sure to test it properly)

if not, look at the integrated backup/restore https://pve.proxmox.com/wiki/Backup_and_Restore