LVM devices still shown in pvesm list after deleting VM

S

schoda

Member
Proxmox Subscriber
May 17, 2019
50
4
8
Hi,

i've deleted two VMs (VMID 199 and 201) and the devices are still there.

root@pm-01:~# pvesm list vg-cluster01-storage01|grep vm-201
vg-cluster01-storage01:vm-201-disk-1 raw 106300440576 201
root@pm-01:~# pvesm list vg-cluster01-storage01|grep vm-199
vg-cluster01-storage01:vm-199-disk-1 raw 106300440576 199

root@pm-01:~# ls /dev/mapper/*201*
/dev/mapper/vg--cluster01--storage01-vm--201--disk--1
root@pm-01:~# ls /dev/mapper/*199*
/dev/mapper/vg--cluster01--storage01-vm--199--disk--1
root@pm-01:~# cd /etc/pve/

The VM config is gone:
root@pm-01:/etc/pve# find . -iname 199.conf
root@pm-01:/etc/pve# find . -iname 201.conf
root@pm-01:/etc/pve#

lsblk:
├─vg--cluster01--storage01-vm--199--disk--1 253:107 0 99G 0 lvm
│ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_5a94a981e87233b582c437aab1b7e55a_tmeta 253:180 0 52M 0 lvm
│ │ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_5a94a981e87233b582c437aab1b7e55a-tpool 253:182 0 10G 0 lvm
│ │ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_5a94a981e87233b582c437aab1b7e55a 253:183 0 10G 0 lvm
│ │ └─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_1159251d5cd1268c47a53f70593b566b 253:184 0 10G 0 lvm
│ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_5a94a981e87233b582c437aab1b7e55a_tdata 253:181 0 10G 0 lvm
│ │ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_5a94a981e87233b582c437aab1b7e55a-tpool 253:182 0 10G 0 lvm
│ │ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_5a94a981e87233b582c437aab1b7e55a 253:183 0 10G 0 lvm
│ │ └─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_1159251d5cd1268c47a53f70593b566b 253:184 0 10G 0 lvm
│ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5_tmeta 253:185 0 256M 0 lvm
│ │ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5-tpool 253:187 0 50G 0 lvm
│ │ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5 253:188 0 50G 0 lvm
│ │ └─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_b48f654cbb990227fe5a4d77844e3ab5 253:189 0 50G 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5_tdata 253:186 0 50G 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5-tpool 253:187 0 50G 0 lvm
│ ├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5 253:188 0 50G 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_b48f654cbb990227fe5a4d77844e3ab5 253:189 0 50G 0 lvm
├─vg--cluster01--storage01-vm--201--disk--1 253:108 0 99G 0 lvm
│ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8ca7e9904657bd7da00182f974341d55_tmeta 253:170 0 52M 0 lvm
│ │ └─vg_7fecd21df3064ad10ae45bab13776110-tp_8ca7e9904657bd7da00182f974341d55-tpool 253:172 0 10G 0 lvm
│ │ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8ca7e9904657bd7da00182f974341d55 253:173 0 10G 0 lvm
│ │ └─vg_7fecd21df3064ad10ae45bab13776110-brick_8ca7e9904657bd7da00182f974341d55 253:174 0 10G 0 lvm
│ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8ca7e9904657bd7da00182f974341d55_tdata 253:171 0 10G 0 lvm
│ │ └─vg_7fecd21df3064ad10ae45bab13776110-tp_8ca7e9904657bd7da00182f974341d55-tpool 253:172 0 10G 0 lvm
│ │ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8ca7e9904657bd7da00182f974341d55 253:173 0 10G 0 lvm
│ │ └─vg_7fecd21df3064ad10ae45bab13776110-brick_8ca7e9904657bd7da00182f974341d55 253:174 0 10G 0 lvm
│ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8cc33cc7b577c0f7c03960a49801ec20_tmeta 253:175 0 256M 0 lvm
│ │ └─vg_7fecd21df3064ad10ae45bab13776110-tp_8cc33cc7b577c0f7c03960a49801ec20-tpool 253:177 0 50G 0 lvm
│ │ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8cc33cc7b577c0f7c03960a49801ec20 253:178 0 50G 0 lvm
│ │ └─vg_7fecd21df3064ad10ae45bab13776110-brick_8cc33cc7b577c0f7c03960a49801ec20 253:179 0 50G 0 lvm
│ └─vg_7fecd21df3064ad10ae45bab13776110-tp_8cc33cc7b577c0f7c03960a49801ec20_tdata 253:176 0 50G 0 lvm
│ └─vg_7fecd21df3064ad10ae45bab13776110-tp_8cc33cc7b577c0f7c03960a49801ec20-tpool 253:177 0 50G 0 lvm
│ ├─vg_7fecd21df3064ad10ae45bab13776110-tp_8cc33cc7b577c0f7c03960a49801ec20 253:178 0 50G 0 lvm
│ └─vg_7fecd21df3064ad10ae45bab13776110-brick_8cc33cc7b577c0f7c03960a49801ec20 253:179 0 50G 0 lvm

In the past it helped to
dmsetup remove <device>

But those two are still in use according to dmsetup remove and they are not configured on any other VM as shared disk.

root@pm-01:/etc/pve# find . -iname ???.conf -exec grep vm-199 {} \;
root@pm-01:/etc/pve# find . -iname ???.conf -exec grep vm-201 {} \;

Those where glusterfs test VMs ...

Whats the best way to get rid of those? pvesm remove vg-cluster01-storage01:vm-201-disk-1?

Thanks & Cheers,
Daniel
 
  • Like
Reactions: neail
Hi,
could you check your task log for the 'VM 199 - Destroy' and 'VM 201 - Destroy' tasks? Maybe there's an error/warning about the failed removal of the disks. You can check with
Code: 
 fuser -vam /dev/mapper/<VM-DISK>
to see whether a process still uses the disk. If you're sure nobody is using the disks anymore, you can try doing 'pvesm remove'.
 
Hi,

sorry for the late response.

fuser -vam /dev/mapper/vg--cluster01--storage01-vm--201--disk--1
USER PID ACCESS COMMAND
/dev/dm-108:

fuser -vam /dev/mapper/vg--cluster01--storage01-vm--199--disk--1
USER PID ACCESS COMMAND
/dev/dm-107:

So they should not be in use by anything.

But i'm unsure about what pvesm remove does. Manpage says:
pvesm remove <storage>

Delete storage configuration.

<storage>: <string>
The storage identifier.

Does this mean it only deletes configuration files?

I got a 100+ VM disks on this volume group i want to keep, i just want to get rid of these two devices...

I searched the log files but i wasn't able to find something with the string you posted. I had a look for destroy only and found this:
/var/log/pve/tasks/index.1:UPID:pm-01:0000B5A7:01BB2200:5DB8679C:qmdestroy:199:root@pam: 5DB8679F OK
/var/log/pve/tasks/index.1:UPID:pm-01:0000B757:01BB2B8B:5DB867B5:qmdestroy:201:root@pam: 5DB867B7 OK

Seems like both tasks finished successfully?

Thanks & Cheers,
Daniel
 
schoda said:
But i'm unsure about what pvesm remove does. Manpage says:
pvesm remove <storage>

Delete storage configuration.

<storage>: <string>
The storage identifier.

Does this mean it only deletes configuration files?
Click to expand...

Sorry I missed this. Yes, 'pvesm remove' is only for the storage configuration. For some reason I thought we had a feature for removing single disks with 'pvesm' as well...
Does
Code: 
lvremove <VG>/vm-<VMID>-disk-<N>
work?
 
Sadly that didn't help.

root@pm-01:~# lvremove /dev/mapper/vg--cluster01--storage01-vm--199--disk--1
Logical volume vg-cluster01-storage01/vm-199-disk-1 is used by another device.
root@pm-01:~# lvremove /dev/mapper/vg--cluster01--storage01-vm--201--disk--1
Logical volume vg-cluster01-storage01/vm-201-disk-1 is used by another device.

According to http://blog.roberthallam.org/2017/12/solved-logical-volume-is-used-by-another-device/comment-page-1/ you need to dmsetup remove before using lvremove. But dmsetup also says busy. I could try --force on dmsetup remove but i'm not sure if that would be the best idea.
 
There are a few other things you could try, described in this article. If the device is really still in use, doing '--force' won't help, because the LVM will still be there. It's just that dmsetup doesn't 'see' it anymore. From the man page:
Open devices cannot be removed, but adding --force will replace the table with one that fails all I/O.
Click to expand...
In case that 'dmsetup' is the only thing blocking 'lvremove' it might work though. But if there were no other blockers, why would 'dmsetup' fail in the first place?
 
fuser and lsof do not show anything accessing those devices. Adding --force didn't help. Same error device busy.

I'm unsure if a reboot of the system would help since it is shared LVM.
 
schoda said:
fuser and lsof do not show anything accessing those devices. Adding --force didn't help. Same error device busy.

I'm unsure if a reboot of the system would help since it is shared LVM.
Click to expand...

From the article I linked to:
If no processes show as accessing the volume in lsof, check to make sure nothing is sharing the filesystem (i.e. rsync, nfs, or samba).
Click to expand...

Maybe the fact that it's shared is the cause of the problem? How exactly did you share it?
 
Fabian_E said:
From the article I linked to:


Maybe the fact that it's shared is the cause of the problem? How exactly did you share it?
Click to expand...

It's a 6 server setup each one is connected to an SAN via FC (multipath). As backend we are using LVM shared. Working like a charm expect lvm in lvm seems to make some trouble sometimes. The affected volume is 30T in size and hosting ~100 VM disks.

I have to a admit that i'm not on the latest 5.4 version or 6.x, currently waiting for free 10G ports in our datacenter.

Those two VMs (199 and 201) where GlusterFS test installation where i played with heketi and docker. It was a three node GlusterFS setup, one VM was deleted just fine. I'm currently working around those two VMIDs simple by "touch 199.conf; touch 201.conf" in qemu-server directory.
 
Last edited:
Does deactivating the logical volume
Code: 
lvchange -an <vg>/<lv>
also throw an error?
 
Fabian_E said:
Does deactivating the logical volume
Code: 
lvchange -an <vg>/<lv>
also throw an error?
Click to expand...

Yes it also throws an error :(

root@pm-01:~# lvchange -an vg-cluster01-storage01/vm-201-disk-1
Logical volume vg-cluster01-storage01/vm-201-disk-1 is used by another device.
root@pm-01:~# lvchange -an vg-cluster01-storage01/vm-199-disk-1
Logical volume vg-cluster01-storage01/vm-199-disk-1 is used by another device.
 
I did a bit of searching in the forum and found two threads that could be related. See first thread and second thread.
Basically we need to use the 'global_filter' setting in '/etc/lvm/lvm.conf' so that the LVM utils ignore /dev/mapper/<vm-image> paths. Hopefully this fixes your problem. Versions lvm2 >= 2.03.02-pve2 include the relevant change (bug report).
 
I'm currently on lvm2 2.02.168-pve6, apt-get dist-upgrade does not show any new lvm2 package. Guess this is Proxmox 6.x?

I played with the global_filter, before it was like
#global_filter = [ "r|/dev/zd.*|", "r|/dev/mapper/pve-.*|", "r|/dev/mapper/vg_.*-brick_.*|", "r|/dev/mapper/vg_.*-tp_.*|" ]
Now i got:
global_filter = [ "r|/dev/zd.*|", "r|/dev/mapper/pve-.*|", "r|/dev/mapper/.*-vm--[0-9]+--disk--[0-9]+|" ]

root@pm-01:~# lvchange -an vg-cluster01-storage01/vm-201-disk-1
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 10737352704: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 10737410048: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 53687025664: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 53687083008: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 10737352704: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 10737410048: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 53687025664: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 53687083008: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 4096: Input/output error
Logical volume vg-cluster01-storage01/vm-201-disk-1 is used by another device.
root@pm-01:~# lvchange -an vg-cluster01-storage01/vm-199-disk-1
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 10737352704: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 10737410048: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8ca7e9904657bd7da00182f974341d55: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 53687025664: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 53687083008: Input/output error
/dev/vg_7fecd21df3064ad10ae45bab13776110/brick_8cc33cc7b577c0f7c03960a49801ec20: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 10737352704: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 10737410048: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_1159251d5cd1268c47a53f70593b566b: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 53687025664: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 53687083008: Input/output error
/dev/vg_4a9d834009d319ddb8e76c38f1ad6800/brick_b48f654cbb990227fe5a4d77844e3ab5: read failed after 0 of 4096 at 4096: Input/output error
Logical volume vg-cluster01-storage01/vm-199-disk-1 is used by another device.

Edit:
Thanks for your help so far. Since i worked around those VMIDs already it's no big deal leaving it as it is for now. I'll try to upgrade to Proxmox 6 as soon as i got my new network interfaces up an running. After the upgrade and hopefully lvm2 2.03 i can remove those devices.
 
Last edited:
I've found the solution, besides checking fuser and lsof you need to check dmsetup info too. If the open count is > 0 check e.g. ls -la /sys/dev/block/251\:19/holders/ like mentioned here https://forum.proxmox.com/threads/lvm-dmsetup-nightmares.58182/. Looking at lsblk might also be a good idea. Mine looked like:

vg--cluster01--storage01-vm--199--disk--1 253:107 0 99G 0 lvm
├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5_tmeta 253:185 0 256M 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5-tpool 253:187 0 50G 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_b48f654cbb990227fe5a4d77844e3ab5 253:189 0 50G 0 lvm
└─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5_tdata 253:186 0 50G 0 lvm
└─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5-tpool 253:187 0 50G 0 lvm
└─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_b48f654cbb990227fe5a4d77844e3ab5 253:189 0 50G 0 lvm

In the end i was able to remove all the devices with dmsetup remove.

Edit: Had to to that on all nodes, now it's clean again.
 
Last edited:
schoda said:
I've found the solution, besides checking fuser and lsof you need to check dmsetup info too. If the open count is > 0 check e.g. ls -la /sys/dev/block/251\:19/holders/ like mentioned here https://forum.proxmox.com/threads/lvm-dmsetup-nightmares.58182/. Looking at lsblk might also be a good idea. Mine looked like:

vg--cluster01--storage01-vm--199--disk--1 253:107 0 99G 0 lvm
├─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5_tmeta 253:185 0 256M 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5-tpool 253:187 0 50G 0 lvm
│ └─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_b48f654cbb990227fe5a4d77844e3ab5 253:189 0 50G 0 lvm
└─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5_tdata 253:186 0 50G 0 lvm
└─vg_4a9d834009d319ddb8e76c38f1ad6800-tp_b48f654cbb990227fe5a4d77844e3ab5-tpool 253:187 0 50G 0 lvm
└─vg_4a9d834009d319ddb8e76c38f1ad6800-brick_b48f654cbb990227fe5a4d77844e3ab5 253:189 0 50G 0 lvm

In the end i was able to remove all the devices with dmsetup remove.

Edit: Had to to that on all nodes, now it's clean again.
Click to expand...

Glad you were able to resolve it. And good to know that lsof isn't enough in such situations.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back