lvm/dmsetup nightmares

[Spiros Pap]

Hi all,

I tried to remove an FC VOL from proxmox and it didn;t go very well.
The sequence was this:
I had moved the disk of a VM to another VOL. After I did that, i tried to delete the unused disk, but proxmox didn't do it with the message:

error with cfs lock 'storage-gix': lvremove 'volgix-3par/vm-7001-disk-0' error: Logical volume volgix-3par/vm-7001-disk-0 is used by another device.

Assuming that the same LV also exists on the new LUN, i did:
dmsetup remove volgix--3par-vm--7001--disk--0

Now, the vgs command does not return on the node. There is no /dev/mapper/volgix--3par-vm--7001--disk--0 of course and the respective device dm-17.


It is not clear to me, why vgs does not return. The VOL is not used by proxmox.
How can i find what creates the deadlock and fix it?

Sp
PS: A reboot is not an option right now.

 

[udo]

Hi all,

I tried to remove an FC VOL from proxmox and it didn;t go very well.
The sequence was this:
I had moved the disk of a VM to another VOL. After I did that, i tried to delete the unused disk, but proxmox didn't do it with the message:

error with cfs lock 'storage-gix': lvremove 'volgix-3par/vm-7001-disk-0' error: Logical volume volgix-3par/vm-7001-disk-0 is used by another device.

Assuming that the same LV also exists on the new LUN, i did:
dmsetup remove volgix--3par-vm--7001--disk--0

Now, the vgs command does not return on the node. There is no /dev/mapper/volgix--3par-vm--7001--disk--0 of course and the respective device dm-17.


It is not clear to me, why vgs does not return. The VOL is not used by proxmox.
How can i find what creates the deadlock and fix it?

Sp
PS: A reboot is not an option right now.

Hi,
if you are sure, that the device are not used, you can follow this guide i wrote for me:

dmsetup info /dev/sata/vm-210-disk-1
Name:              sata-vm--210--disk--1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        6
Event number:      0
Major, minor:      251, 19
Number of targets: 4
UUID: LVM-L1Tx4VW6xrUkd4PguEazxvvdEb3B603avNg7RO8pNx8uChKq0aRw4GSdwpyTdNG

## Open count > 0 is the issue!!

# dmsetup info -c | grep 210
sata-vm--210--disk--1     251  19 L--w    6    1      0 LVM-L1Tx4VW6xrUkd4PguEazxvvdEb3B603avNg7RO8pNx8uChKq0aRw4GSdwpyTdNGc      

# ls -la /sys/dev/block/251\:19/holders/
total 0
drwxr-xr-x 2 root root 0 Mar 23 19:49 .
drwxr-xr-x 9 root root 0 Mar  4 09:40 ..
lrwxrwxrwx 1 root root 0 Mar  4 09:42 dm-46 -> ../../dm-46
lrwxrwxrwx 1 root root 0 Mar  4 09:42 dm-47 -> ../../dm-47
lrwxrwxrwx 1 root root 0 Mar  4 09:42 dm-48 -> ../../dm-48
lrwxrwxrwx 1 root root 0 Mar  4 09:42 dm-49 -> ../../dm-49
lrwxrwxrwx 1 root root 0 Mar  4 09:42 dm-50 -> ../../dm-50
lrwxrwxrwx 1 root root 0 Mar  4 09:42 dm-51 -> ../../dm-51


# dmsetup remove /dev/dm-46
# dmsetup remove /dev/dm-47
# dmsetup remove /dev/dm-48
# dmsetup remove /dev/dm-49
# dmsetup remove /dev/dm-50
# dmsetup remove /dev/dm-51

# dmsetup info /dev/sata/vm-210-disk-1
Name:              sata-vm--210--disk--1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      251, 19
Number of targets: 1
UUID: LVM-L1Tx4VW6xrUkd4PguEazxvvdEb3B603avNg7RO8pNx8uChKq0aRw4GSdwpyTdNGc

# lvchange -a n /dev/sata/vm-210-disk-1

Udo

 

[Stoiko Ivanov]

hm - apart from @udo 's comment and script:
* vgs blocking for some time could be due to having LVM's inside guests, which by chance have the same name (e.g. because the OS installer always names them the same way).
* unless you don't explicitly make sure that the disk-images of your guests are not scanned by the hosts lvm-utils (adapt /etc/lvm/lvm.conf - the global_filter setting) - all the VGs inside the guests are also activated on your PVE-node
* this would also explain why your image is used by another device (the image contains a PV, which is imported in the node - and all LV's device-nodes are users of it)

I hope this helps!

 

[Spiros Pap]

Hi Udo, Stoiko and all,

I really appreciate your help. Udo's guide is really useful!
I managed to resolve the problem, before Udo's reply, so unfortunately I didn't followed Udo's steps.
Anyway, my problem device was (from my terminal history):
root@hs4:~# dmsetup info volgix--3par-vm--7001--disk--0
Name: volgix--3par-vm--7001--disk--0
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 2
Event number: 0
Major, minor: 253, 24
Number of targets: 2
UUID: LVM-eYqsvgNruGhQvVLVSWGvOC3SR0hRE1u1jrP5r86g7k7JO6dqKhC9lUyos7iYirYA

The Open count of 2, were two LVs inside the guest (as Stoiko suggested). I had actually removed these two devices (dmsetup removed) and after that I did the dmsetup remove volgix--3par-vm--7001--disk--0. Then it got stuck.

The 2 targets that volgix--3par-vm--7001--disk--0 had, were two multipathed LUNs stripped together (raid0), that were supposedly active and healthy, all the time.So, having no clue on what to do, I tried to remove them from multipathd (blacklisted them) to make them disappear but they didn't! Then I did a multipath -f for each one of them and after that vgs worked again....and everything came back to life.

It is not really clear to me, why vgs got stuck in the first place, since volgix--3par-vm--7001--disk--0 should have an open count of 0, when i removed it.
It is not clear either, why it got unstuck when i flushed/removed the target devices underneath.

Anyway, lvm/dmsetup is a wonderful beast that can bite you. While, I don't get why on earth the lvmutils, get stuck, so easily, I understand that the Proxmox team can't do much about it. But since LVM is such a basic and common element of Proxmox then it would worth handling such issues a bit better. The GUI was working on the problematic node but no info could be retrieved for the VMs and actions on VMs were crippled or not working at all. Worst of all, was that I could not migrate the VMs to another node (is the VM migration really correlated with LVM?).

Thanx again for the info!

Sp

 

[Stoiko Ivanov]

The Open count of 2, were two LVs inside the guest (as Stoiko suggested). I had actually removed these two devices (dmsetup removed) and after that I did the dmsetup remove volgix--3par-vm--7001--disk--0. Then it got stuck.

Please try to add /dev/volgix-3par/.* to the global_filter in /etc/lvm/lvm.conf

that usually helps in those cases