LVM: devices in /dev/mapper/ still shown but no corresponding VM with that ID

S

schoda

Member
Proxmox Subscriber
May 17, 2019
50
4
8
Hi,

TL;DR: No VM with ID 171 present but raw devices in /dev/mapper still shown

Code: 
~$ ssh root@pm-01 qm config 171
Configuration file 'nodes/pm-01/qemu-server/171.conf' does not exist
~$ ssh root@pm-02 qm config 171
Configuration file 'nodes/pm-02/qemu-server/171.conf' does not exist
~$ ssh root@pm-03 qm config 171
Configuration file 'nodes/pm-03/qemu-server/171.conf' does not exist
~$ ssh root@pm-04 qm config 171
Configuration file 'nodes/pm-04/qemu-server/171.conf' does not exist
~$ ssh root@pm-05 qm config 171
Configuration file 'nodes/pm-05/qemu-server/171.conf' does not exist
~$ ssh root@pm-06 qm config 171
Configuration file 'nodes/pm-06/qemu-server/171.conf' does not exist

~$ ssh root@pm-01 lvs|grep 171
~$ ssh root@pm-02 lvs|grep 171
~$ ssh root@pm-03 lvs|grep 171
~$ ssh root@pm-04 lvs|grep 171
~$ ssh root@pm-05 lvs|grep 171
~$ ssh root@pm-06 lvs|grep 171

~$ ssh root@pm-06 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0
/dev/mapper/vg--cluster01--s4h-vm--171--disk--1
~$ ssh root@pm-05 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
~$ ssh root@pm-04 ls /dev/mapper/*171*
ls: cannot access '/dev/mapper/*171*': No such file or directory
~$ ssh root@pm-03 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
~$ ssh root@pm-02 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
$ ssh root@pm-01 ls /dev/mapper/*171*
/dev/mapper/vg—cluster01--storage01-vm--171--disk--0

There is no VM with VM ID 171 but the raw devices are still present in /dev/mapper. Strangely enough on two different Volumes. I stumpeld across this when trying to create a new VM and it automatically filled in 171 as VM ID. I'm able to create new VMs with another ID.

lvremove does not work:
root@pm-01:~# lvremove /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
Failed to find logical volume "vg-cluster01-storage01/vm-171-disk-0"

Thanks & Cheers,
Daniel
 
Last edited:
Could you please post the complete output of LVS, VG? Where does the LVM storage reside?
 
Code: 
root@pm-01:~# lvs
  LV                    VG                               Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data                  pve                              twi-a-tz--  75,87g             0,00   0,04                         
  root                  pve                              -wi-ao----  33,75g                                                 
  swap                  pve                              -wi-ao----   8,00g                                                 
  lv-cluster01-backup01 vg-cluster01-backup01            -wi-ao---- 900,00g                                                 
  vm-161-disk-0         vg-cluster01-XYZ            -wi-a-----   1,00t                                                 
  vm-164-disk-0         vg-cluster01-XYZ            -wi-a-----   1,00t                                                 
  vm-167-disk-0         vg-cluster01-YXZ              -wi------- 500,00g                                                 
  vm-168-disk-0         vg-cluster01-YXZ              -wi------- 500,00g                                                 
  vm-169-disk-0         vg-cluster01-online              -wi------- 750,00g                                                 
  vm-179-disk-0         vg-cluster01-s4h                 -wi------- 500,00g                                                 
  vm-179-disk-1         vg-cluster01-s4h                 -wi-------  50,00g                                                 
  vm-183-disk-0         vg-cluster01-s4h                 -wi-------   2,05t                                                 
  vm-183-disk-1         vg-cluster01-s4h                 -wi------- 100,00g                                                 
  vm-183-disk-2         vg-cluster01-s4h                 -wi------- 100,00g                                                 
  vm-104-disk-0         vg-cluster01-storage01           -wi------- 110,00g                                                 
  vm-105-disk-0         vg-cluster01-storage01           -wi-a-----   1,00t                                                 
  vm-105-disk-1         vg-cluster01-storage01           -wi-a----- 300,00g                                                 
  vm-106-disk-0         vg-cluster01-storage01           -wi-a----- 510,00g                                                 
  vm-107-disk-0         vg-cluster01-storage01           -wi-ao---- 110,00g                                                 
  vm-108-disk-0         vg-cluster01-storage01           -wi-a----- 110,00g                                                 
  vm-109-disk-0         vg-cluster01-storage01           -wi-a----- 510,00g                                                 
  vm-110-disk-0         vg-cluster01-storage01           -wi-a----- 210,00g                                                 
  vm-111-disk-0         vg-cluster01-storage01           -wi-a----- 210,00g                                                 
  vm-112-disk-0         vg-cluster01-storage01           -wi-ao----  41,00g                                                 
  vm-113-disk-0         vg-cluster01-storage01           -wi-a-----  41,00g                                                 
  vm-114-disk-0         vg-cluster01-storage01           -wi-ao----  63,00g                                                 
  vm-115-disk-0         vg-cluster01-storage01           -wi-a----- 109,00g                                                 
  vm-116-disk-0         vg-cluster01-storage01           -wi-a----- 210,00g                                                 
  vm-117-disk-0         vg-cluster01-storage01           -wi-ao---- 105,00g                                                 
  vm-118-disk-0         vg-cluster01-storage01           -wi-a----- 105,00g                                                 
  vm-119-disk-0         vg-cluster01-storage01           -wi-a-----  20,00g                                                 
  vm-120-disk-0         vg-cluster01-storage01           -wi-a----- 130,00g                                                 
  vm-121-disk-0         vg-cluster01-storage01           -wi-a----- 505,00g                                                 
  vm-122-disk-0         vg-cluster01-storage01           -wi-a----- 505,00g                                                 
  vm-123-disk-0         vg-cluster01-storage01           -wi-ao---- 205,00g                                                 
  vm-124-disk-0         vg-cluster01-storage01           -wi-ao---- 105,00g                                                 
  vm-125-disk-0         vg-cluster01-storage01           -wi-ao----  51,00g                                                 
  vm-127-disk-0         vg-cluster01-storage01           -wi-a----- 685,00g                                                 
  vm-128-disk-0         vg-cluster01-storage01           -wi-ao---- 505,00g                                                 
  vm-129-disk-0         vg-cluster01-storage01           -wi-ao----  55,00g                                                 
  vm-129-disk-1         vg-cluster01-storage01           -wi-ao---- 205,00g                                                 
  vm-129-disk-2         vg-cluster01-storage01           -wi-ao---- 255,00g                                                 
  vm-130-disk-0         vg-cluster01-storage01           -wi-a-----  55,00g                                                 
  vm-131-disk-0         vg-cluster01-storage01           -wi-ao---- 160,00g                                                 
  vm-132-disk-0         vg-cluster01-storage01           -wi-a----- 155,00g                                                 
  vm-133-disk-0         vg-cluster01-storage01           -wi-ao----  12,00g                                                 
  vm-134-disk-0         vg-cluster01-storage01           -wi-ao---- 260,00g                                                 
  vm-135-disk-0         vg-cluster01-storage01           -wi-a----- 260,00g                                                 
  vm-136-disk-0         vg-cluster01-storage01           -wi-a----- 110,00g                                                 
  vm-137-disk-0         vg-cluster01-storage01           -wi-a----- 255,00g                                                 
  vm-138-disk-0         vg-cluster01-storage01           -wi-a----- 805,00g                                                 
  vm-139-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-140-disk-0         vg-cluster01-storage01           -wi-a----- 125,00g                                                 
  vm-141-disk-0         vg-cluster01-storage01           -wi-a----- 105,00g                                                 
  vm-142-disk-0         vg-cluster01-storage01           -wi-a----- 250,00g                                                 
  vm-143-disk-0         vg-cluster01-storage01           -wi-ao---- 250,00g                                                 
  vm-144-disk-0         vg-cluster01-storage01           -wi-ao----  17,00g                                                 
  vm-145-disk-0         vg-cluster01-storage01           -wi-a----- 500,00g                                                 
  vm-146-disk-0         vg-cluster01-storage01           -wi-ao---- 505,00g                                                 
  vm-147-disk-0         vg-cluster01-storage01           -wi-ao---- 152,00g                                                 
  vm-148-disk-0         vg-cluster01-storage01           -wi-ao---- 202,00g                                                 
  vm-149-disk-0         vg-cluster01-storage01           -wi-a-----  52,00g                                                 
  vm-150-disk-0         vg-cluster01-storage01           -wi-a-----  52,00g                                                 
  vm-151-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-152-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-153-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-154-disk-0         vg-cluster01-storage01           -wi-a-----  52,00g                                                 
  vm-155-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-156-disk-0         vg-cluster01-storage01           -wi-a----- 122,00g                                                 
  vm-157-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-158-disk-0         vg-cluster01-storage01           -wi-a----- 122,00g                                                 
  vm-159-disk-0         vg-cluster01-storage01           -wi-a----- 502,00g                                                 
  vm-160-disk-0         vg-cluster01-storage01           -wi-ao----  76,00g                                                 
  vm-162-disk-0         vg-cluster01-storage01           -wi-a-----  32,00g                                                 
  vm-163-disk-0         vg-cluster01-storage01           -wi-ao----  52,00g                                                 
  vm-165-disk-0         vg-cluster01-storage01           -wi-ao----  27,00g                                                 
  vm-165-disk-1         vg-cluster01-storage01           -wi-ao----  81,00g                                                 
  vm-166-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-170-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-172-disk-0         vg-cluster01-storage01           -wi-a-----  41,00g                                                 
  vm-173-disk-0         vg-cluster01-storage01           -wi-a----- 201,00g                                                 
  vm-174-disk-0         vg-cluster01-storage01           -wi-ao---- 200,00g                                                 
  vm-175-disk-0         vg-cluster01-storage01           -wi-a-----  32,00g                                                 
  vm-176-disk-0         vg-cluster01-storage01           -wi-a----- 201,00g                                                 
  vm-177-disk-0         vg-cluster01-storage01           -wi-ao---- 102,00g                                                 
  vm-178-disk-0         vg-cluster01-storage01           -wi-ao---- 201,00g                                                 
  vm-180-disk-0         vg-cluster01-storage01           -wi-------  50,00g                                                 
  vm-184-disk-0         vg-cluster01-storage01           -wi-ao---- 305,00g                                                 
  vm-185-disk-0         vg-cluster01-storage01           -wi-ao----  20,00g                                                 
  vm-185-disk-1         vg-cluster01-storage01           -wi-ao----  50,00g                                                 
  vm-186-disk-0         vg-cluster01-storage01           -wi-ao----  20,00g                                                 
  vm-186-disk-1         vg-cluster01-storage01           -wi-ao----  50,00g                                                 
  vm-100-disk-0         vg-cluster01-storage01-ZYX -wi-a-----   1,00t                                                 
  vm-101-disk-0         vg-cluster01-storage01-ZYX -wi-ao----   1,00t                                                 
  vm-102-disk-0         vg-cluster01-storage01-ZYX -wi-a-----   1,00t                                                 
  vm-103-disk-0         vg-cluster01-storage01-ZYX -wi-a-----   1,00t                                                 
  vm-126-disk-0         vg-cluster01-storage01-ZYX -wi-a----- 200,00g                                                 
  lv-isotemplatestorage vg-isotemplatestorage            -wi-ao---- 190,00g                                                 
root@pm-01:~# pvs
  PV                                           VG                               Fmt  Attr PSize    PFree
  /dev/mapper/pm-cluster01-backup01            vg-cluster01-backup01            lvm2 a--  1024,00g 124,00g
  /dev/mapper/pm-cluster01-XYZ            vg-cluster01-XYZ            lvm2 a--     5,00t   3,00t
  /dev/mapper/pm-cluster01-demovolume          vg-cluster01-demovolume          lvm2 a--   200,00g 200,00g
  /dev/mapper/pm-cluster01-YXZ              vg-cluster01-YXZ              lvm2 a--     1,50t 536,00g
  /dev/mapper/pm-cluster01-online              vg-cluster01-online              lvm2 a--     1,50t 786,00g
  /dev/mapper/pm-cluster01-s4h                 vg-cluster01-s4h                 lvm2 a--     3,00t 222,00g
  /dev/mapper/pm-cluster01-storage01           vg-cluster01-storage01           lvm2 a--    20,00t   5,32t
  /dev/mapper/pm-cluster01-storage01-ZYX vg-cluster01-storage01-ZYX lvm2 a--     5,00t 824,00g
  /dev/mapper/pm-isotemplatestorage            vg-isotemplatestorage            lvm2 a--  1024,00g 834,00g
  /dev/sda3                                    pve                              lvm2 a--   135,62g  16,00g
root@pm-01:~# vgs
  VG                               #PV #LV #SN Attr   VSize    VFree
  pve                                1   3   0 wz--n-  135,62g  16,00g
  vg-cluster01-backup01              1   1   0 wz--n- 1024,00g 124,00g
  vg-cluster01-XYZ              1   2   0 wz--n-    5,00t   3,00t
  vg-cluster01-demovolume            1   0   0 wz--n-  200,00g 200,00g
  vg-cluster01-YXZ                1   2   0 wz--n-    1,50t 536,00g
  vg-cluster01-online                1   1   0 wz--n-    1,50t 786,00g
  vg-cluster01-s4h                   1   5   0 wz--n-    3,00t 222,00g
  vg-cluster01-storage01             1  78   0 wz--n-   20,00t   5,32t
  vg-cluster01-storage01-ZYX   1   5   0 wz--n-    5,00t 824,00g
  vg-isotemplatestorage              1   1   0 wz--n- 1024,00g 834,00g
root@pm-01:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0

My Storage is a Purestorage connected via FC with multipath.

It is working fine besides i have a stale device in /dev/mapper i can't get rid of.

root@pm-06:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0
/dev/mapper/vg--cluster01--s4h-vm--171--disk--1

root@pm-01:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
 
Code: 
dmsetup info /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
dmsetup deps /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
What output do you get with the above commands? And where is the PV 'pm-*' located on (local/storage box)?
 
Code: 
root@pm-01:~# dmsetup info /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
Name:              vg--cluster01--storage01-vm--171--disk--0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 72
Number of targets: 1
UUID: LVM-uU4XEQ45hk00hlUS9tNVO2b2moEE5kDLYYrdhwxfaZa0EyhfIEx2k69HGGddVG2k

root@pm-01:~# dmsetup deps /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
1 dependencies    : (253, 5)

root@pm-06:~# dmsetup info /dev/mapper/*171*
Name:              vg--cluster01--s4h-vm--171--disk--0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 93
Number of targets: 1
UUID: LVM-I0M2R2I3ibpyxr9CrynMAPO94E2HQTbBm1J06TfmX0GPlqf3U0ty2dzo6EQLNdrz

Name:              vg--cluster01--s4h-vm--171--disk--1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 94
Number of targets: 1
UUID: LVM-I0M2R2I3ibpyxr9CrynMAPO94E2HQTbBjU8sacmed1ckfZOwhOrRlY2fpGlJDeCF

root@pm-06:~# dmsetup deps /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0: 1 dependencies   : (253, 1)
/dev/mapper/vg--cluster01--s4h-vm--171--disk--1: 1 dependencies   : (253, 1)

All /dev/mapper/pm* are on a storage box.
 
And where are the PV located on (local / storage box)? With 'lsblk' you could check on the node, if there is a device with matching 'major, minor' number.
 
All my pv used for virtual machines are on a SAN which is connected via Fiber Channel to the proxmox cluster.

First Cluster node (pm-01):
Code: 
sdg                                                        8:96   0    20T  0 disk
└─pm-cluster01-storage01                                 253:5    0    20T  0 mpath
  ├─vg--cluster01--storage01-vm--105--disk--0            253:7    0     1T  0 lvm
  ├─vg--cluster01--storage01-vm--105--disk--1            253:8    0   300G  0 lvm
  ├─vg--cluster01--storage01-vm--106--disk--0            253:9    0   510G  0 lvm
  ├─vg--cluster01--storage01-vm--107--disk--0            253:10   0   110G  0 lvm
  ├─vg--cluster01--storage01-vm--108--disk--0            253:11   0   110G  0 lvm
  ├─vg--cluster01--storage01-vm--109--disk--0            253:12   0   510G  0 lvm
[cut]
  ├─vg--cluster01--storage01-vm--168--disk--0            253:69   0   202G  0 lvm
  ├─vg--cluster01--storage01-vm--169--disk--0            253:70   0   202G  0 lvm
  ├─vg--cluster01--storage01-vm--170--disk--0            253:71   0   102G  0 lvm
  ├─vg--cluster01--storage01-vm--171--disk--0            253:72   0   202G  0 lvm
  ├─vg--cluster01--storage01-vm--172--disk--0            253:73   0    41G  0 lvm
  ├─vg--cluster01--storage01-vm--173--disk--0            253:74   0   201G  0 lvm
[cut]

root@pm-01:~# lvs|grep 171

root@pm-01:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0

The VM with ID 171 does not exist anymore.

Sixth cluster node: (pm-06)
Code: 
sde                                                        8:64   0     3T  0 disk
└─pm-cluster01-s4h                                       253:1    0     3T  0 mpath
  ├─vg--cluster01--s4h-vm--183--disk--0                  253:88   0   2,1T  0 lvm
  ├─vg--cluster01--s4h-vm--183--disk--1                  253:89   0   100G  0 lvm
  ├─vg--cluster01--s4h-vm--183--disk--2                  253:90   0   100G  0 lvm
  ├─vg--cluster01--s4h-vm--179--disk--0                  253:91   0   500G  0 lvm
  ├─vg--cluster01--s4h-vm--179--disk--1                  253:92   0    50G  0 lvm
  ├─vg--cluster01--s4h-vm--171--disk--0                  253:93   0    20G  0 lvm
  └─vg--cluster01--s4h-vm--171--disk--1                  253:94   0    32G  0 lvm
root@pm-06:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0 /dev/mapper/vg--cluster01--s4h-vm--171--disk--1
root@pm-06:~# lvs|grep 171
root@pm-06:~#

We do not have a VM with ID 171 on our cluster anymore. It got deleted (can't say when). When i tried to create a new VM on node 1 (pm-01) via WebUI it picked the id 171 automatically and tried to create the LV. It failed.

Code: 
 device-mapper: create ioctl on vg--cluster01--storage01-vm--171--disk--0LVM-uU4XEQ45hk00hlUS9tNVO2b2moEE5kDLengfrbdk3A8uDm9cbBbFpFQ5k4pgbfhH failed: Device or resource busy
TASK ERROR: unable to create VM 171 - error with cfs lock 'storage-vg-cluster01-storage01': lvcreate 'vg-cluster01-storage01/pve-vm-171' error: Failed to activate new LV.

I had a look around and i found /dev/mapper/vg--cluster01--storage01-vm--171--disk--0 on node 1.
And i found /dev/mapper/vg--cluster01--s4h-vm--171--disk--0 and /dev/mapper/vg--cluster01--s4h-vm--171--disk--1 on node 5/6 (these are two differnet volumes connected from the SAN).
Which do my understanding should not be possible since IDs in proxmox are uniq.

I am currently working around VM ID 171 simply by "touch 171.conf" in /etc/pve/nodes/pm-01/qemu-server/. Now Proxmox uses the next free ID.
 
Last edited:
Code: 
sdg                                                        8:96   0    20T  0 disk

└─pm-cluster01-storage01                                 253:5    0    20T  0 mpath
  ├─vg--cluster01--storage01-vm--171--disk--0            253:72   0   202G  0 lvm

Code: 
sde                                                        8:64   0     3T  0 disk
└─pm-cluster01-s4h                                       253:1    0     3T  0 mpath
  ├─vg--cluster01--s4h-vm--171--disk--0                  253:93   0    20G  0 lvm
  └─vg--cluster01--s4h-vm--171--disk--1                  253:94   0    32G  0 lvm
The VMID is unique, but the disk numbering starts at 0 on each storage.

Does a lvscan show them too? If the VM doesn't exist, then you could try to use the dmsetup to remove the mapping or reboot.
 
  • Like
Reactions: schoda
You must log in or register to reply here.
Top Bottom