[SOLVED] LVM (over iscsi) volume stuck and cannot be created

gradinaruvasile

Active Member
Oct 22, 2015
66
9
28
We have a 3 node cluster and a SAN that is connected via iscsi (using lvm over iscsi).
I had to do some maintenance and move machines around and i noticed that one specific machine could not be migrated between nodes because of errors related to it's storage.

Initially it gave me some errors when i wanted to migrate the machine to another node, along the lines of

Code:
device-mapper: create ioctl on san_lvm_volgroup-vm--135--disk--0 LVM-6kXN8w5HcxnsGojMrnMBI1IaPJ0kS3YSQRpboe0uwleV3UNQq8mJOlkrSDupKSi3 failed: Device or resource busy

I migrated the disk to another storage (one cluster member's nfs shared disk), it went ok.
Now i wanted to migrate it back to the SAN and i still get errors like this:

Code:
create full clone of drive scsi0 (srv005:135/vm-135-disk-0.qcow2)
  device-mapper: create ioctl on san_lvm_volgroup-vm--135--disk--0 LVM-6kXN8w5HcxnsGojMrnMBI1IaPJ0kS3YSQRpboe0uwleV3UNQq8mJOlkrSDupKSi3 failed: Device or resource busy
TASK ERROR: storage migration failed: error with cfs lock 'storage-zesan-lvm': lvcreate 'san_lvm_volgroup/vm-135-disk-0' error:   Failed to activate new LV san_lvm_volgroup/vm-135-disk-0.

lvscan and lvdisplay does not show "vm-135-disk-0". I suppose something is stuck somewhere but i have no idea. I migrated other machines' disks to and from the SAN LVM succesfully.

Where should i start debugging this?
 

Dominic

Proxmox Staff Member
Staff member
Mar 18, 2019
788
71
28
Hi!

Have you been able to solve your problem already? If not, what version of Proxmox VE are you using pveversion -v?
 

gradinaruvasile

Active Member
Oct 22, 2015
66
9
28
Hi,
Still get the same error. I moved the drive to another storage for now. The issue seems to be related to iscsi having something stuck somewhere.

Pveversion (we use the enterprise repos):

Code:
# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.13-3-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-3
pve-kernel-helper: 6.1-3
pve-kernel-5.0: 6.0-11
pve-kernel-4.15: 5.4-7
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-11
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-4
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
 

gradinaruvasile

Active Member
Oct 22, 2015
66
9
28
I managed to move the VM to another node, move the disk to the SAN. Then i could move the VM to a third node.
But i still get this error when i try to move it back to the original node i encountered the error on.
So that server is the only one with these issues.
 

Dominic

Proxmox Staff Member
Staff member
Mar 18, 2019
788
71
28
Please check out if dmsetup can help you, dmsetup table would be a start, for example. Two possibly related threads: 1 & 2.
 

gradinaruvasile

Active Member
Oct 22, 2015
66
9
28
Thanks.
This worked.

Edit - some details:
Basically i identified the stuck device with
Code:
dmsetup table |grep 135
Then removed it with
Code:
dmsetup remove san_lvm_volgroup-vm--135--disk--0
 
Last edited:
  • Like
Reactions: Dominic

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!