LVM: devices in /dev/mapper/ still shown but no corresponding VM with that ID

S

schoda

Member
Proxmox Subscriber
May 17, 2019
50
4
8
Hi,

TL;DR: No VM with ID 171 present but raw devices in /dev/mapper still shown

Code: 
~$ ssh root@pm-01 qm config 171
Configuration file 'nodes/pm-01/qemu-server/171.conf' does not exist
~$ ssh root@pm-02 qm config 171
Configuration file 'nodes/pm-02/qemu-server/171.conf' does not exist
~$ ssh root@pm-03 qm config 171
Configuration file 'nodes/pm-03/qemu-server/171.conf' does not exist
~$ ssh root@pm-04 qm config 171
Configuration file 'nodes/pm-04/qemu-server/171.conf' does not exist
~$ ssh root@pm-05 qm config 171
Configuration file 'nodes/pm-05/qemu-server/171.conf' does not exist
~$ ssh root@pm-06 qm config 171
Configuration file 'nodes/pm-06/qemu-server/171.conf' does not exist

~$ ssh root@pm-01 lvs|grep 171
~$ ssh root@pm-02 lvs|grep 171
~$ ssh root@pm-03 lvs|grep 171
~$ ssh root@pm-04 lvs|grep 171
~$ ssh root@pm-05 lvs|grep 171
~$ ssh root@pm-06 lvs|grep 171

~$ ssh root@pm-06 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0
/dev/mapper/vg--cluster01--s4h-vm--171--disk--1
~$ ssh root@pm-05 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
~$ ssh root@pm-04 ls /dev/mapper/*171*
ls: cannot access '/dev/mapper/*171*': No such file or directory
~$ ssh root@pm-03 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
~$ ssh root@pm-02 ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
$ ssh root@pm-01 ls /dev/mapper/*171*
/dev/mapper/vg—cluster01--storage01-vm--171--disk--0

There is no VM with VM ID 171 but the raw devices are still present in /dev/mapper. Strangely enough on two different Volumes. I stumpeld across this when trying to create a new VM and it automatically filled in 171 as VM ID. I'm able to create new VMs with another ID.

lvremove does not work:
root@pm-01:~# lvremove /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
Failed to find logical volume "vg-cluster01-storage01/vm-171-disk-0"

Thanks & Cheers,
Daniel
 
Last edited:
Could you please post the complete output of LVS, VG? Where does the LVM storage reside?
 
Code: 
root@pm-01:~# lvs
  LV                    VG                               Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data                  pve                              twi-a-tz--  75,87g             0,00   0,04                         
  root                  pve                              -wi-ao----  33,75g                                                 
  swap                  pve                              -wi-ao----   8,00g                                                 
  lv-cluster01-backup01 vg-cluster01-backup01            -wi-ao---- 900,00g                                                 
  vm-161-disk-0         vg-cluster01-XYZ            -wi-a-----   1,00t                                                 
  vm-164-disk-0         vg-cluster01-XYZ            -wi-a-----   1,00t                                                 
  vm-167-disk-0         vg-cluster01-YXZ              -wi------- 500,00g                                                 
  vm-168-disk-0         vg-cluster01-YXZ              -wi------- 500,00g                                                 
  vm-169-disk-0         vg-cluster01-online              -wi------- 750,00g                                                 
  vm-179-disk-0         vg-cluster01-s4h                 -wi------- 500,00g                                                 
  vm-179-disk-1         vg-cluster01-s4h                 -wi-------  50,00g                                                 
  vm-183-disk-0         vg-cluster01-s4h                 -wi-------   2,05t                                                 
  vm-183-disk-1         vg-cluster01-s4h                 -wi------- 100,00g                                                 
  vm-183-disk-2         vg-cluster01-s4h                 -wi------- 100,00g                                                 
  vm-104-disk-0         vg-cluster01-storage01           -wi------- 110,00g                                                 
  vm-105-disk-0         vg-cluster01-storage01           -wi-a-----   1,00t                                                 
  vm-105-disk-1         vg-cluster01-storage01           -wi-a----- 300,00g                                                 
  vm-106-disk-0         vg-cluster01-storage01           -wi-a----- 510,00g                                                 
  vm-107-disk-0         vg-cluster01-storage01           -wi-ao---- 110,00g                                                 
  vm-108-disk-0         vg-cluster01-storage01           -wi-a----- 110,00g                                                 
  vm-109-disk-0         vg-cluster01-storage01           -wi-a----- 510,00g                                                 
  vm-110-disk-0         vg-cluster01-storage01           -wi-a----- 210,00g                                                 
  vm-111-disk-0         vg-cluster01-storage01           -wi-a----- 210,00g                                                 
  vm-112-disk-0         vg-cluster01-storage01           -wi-ao----  41,00g                                                 
  vm-113-disk-0         vg-cluster01-storage01           -wi-a-----  41,00g                                                 
  vm-114-disk-0         vg-cluster01-storage01           -wi-ao----  63,00g                                                 
  vm-115-disk-0         vg-cluster01-storage01           -wi-a----- 109,00g                                                 
  vm-116-disk-0         vg-cluster01-storage01           -wi-a----- 210,00g                                                 
  vm-117-disk-0         vg-cluster01-storage01           -wi-ao---- 105,00g                                                 
  vm-118-disk-0         vg-cluster01-storage01           -wi-a----- 105,00g                                                 
  vm-119-disk-0         vg-cluster01-storage01           -wi-a-----  20,00g                                                 
  vm-120-disk-0         vg-cluster01-storage01           -wi-a----- 130,00g                                                 
  vm-121-disk-0         vg-cluster01-storage01           -wi-a----- 505,00g                                                 
  vm-122-disk-0         vg-cluster01-storage01           -wi-a----- 505,00g                                                 
  vm-123-disk-0         vg-cluster01-storage01           -wi-ao---- 205,00g                                                 
  vm-124-disk-0         vg-cluster01-storage01           -wi-ao---- 105,00g                                                 
  vm-125-disk-0         vg-cluster01-storage01           -wi-ao----  51,00g                                                 
  vm-127-disk-0         vg-cluster01-storage01           -wi-a----- 685,00g                                                 
  vm-128-disk-0         vg-cluster01-storage01           -wi-ao---- 505,00g                                                 
  vm-129-disk-0         vg-cluster01-storage01           -wi-ao----  55,00g                                                 
  vm-129-disk-1         vg-cluster01-storage01           -wi-ao---- 205,00g                                                 
  vm-129-disk-2         vg-cluster01-storage01           -wi-ao---- 255,00g                                                 
  vm-130-disk-0         vg-cluster01-storage01           -wi-a-----  55,00g                                                 
  vm-131-disk-0         vg-cluster01-storage01           -wi-ao---- 160,00g                                                 
  vm-132-disk-0         vg-cluster01-storage01           -wi-a----- 155,00g                                                 
  vm-133-disk-0         vg-cluster01-storage01           -wi-ao----  12,00g                                                 
  vm-134-disk-0         vg-cluster01-storage01           -wi-ao---- 260,00g                                                 
  vm-135-disk-0         vg-cluster01-storage01           -wi-a----- 260,00g                                                 
  vm-136-disk-0         vg-cluster01-storage01           -wi-a----- 110,00g                                                 
  vm-137-disk-0         vg-cluster01-storage01           -wi-a----- 255,00g                                                 
  vm-138-disk-0         vg-cluster01-storage01           -wi-a----- 805,00g                                                 
  vm-139-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-140-disk-0         vg-cluster01-storage01           -wi-a----- 125,00g                                                 
  vm-141-disk-0         vg-cluster01-storage01           -wi-a----- 105,00g                                                 
  vm-142-disk-0         vg-cluster01-storage01           -wi-a----- 250,00g                                                 
  vm-143-disk-0         vg-cluster01-storage01           -wi-ao---- 250,00g                                                 
  vm-144-disk-0         vg-cluster01-storage01           -wi-ao----  17,00g                                                 
  vm-145-disk-0         vg-cluster01-storage01           -wi-a----- 500,00g                                                 
  vm-146-disk-0         vg-cluster01-storage01           -wi-ao---- 505,00g                                                 
  vm-147-disk-0         vg-cluster01-storage01           -wi-ao---- 152,00g                                                 
  vm-148-disk-0         vg-cluster01-storage01           -wi-ao---- 202,00g                                                 
  vm-149-disk-0         vg-cluster01-storage01           -wi-a-----  52,00g                                                 
  vm-150-disk-0         vg-cluster01-storage01           -wi-a-----  52,00g                                                 
  vm-151-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-152-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-153-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-154-disk-0         vg-cluster01-storage01           -wi-a-----  52,00g                                                 
  vm-155-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-156-disk-0         vg-cluster01-storage01           -wi-a----- 122,00g                                                 
  vm-157-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-158-disk-0         vg-cluster01-storage01           -wi-a----- 122,00g                                                 
  vm-159-disk-0         vg-cluster01-storage01           -wi-a----- 502,00g                                                 
  vm-160-disk-0         vg-cluster01-storage01           -wi-ao----  76,00g                                                 
  vm-162-disk-0         vg-cluster01-storage01           -wi-a-----  32,00g                                                 
  vm-163-disk-0         vg-cluster01-storage01           -wi-ao----  52,00g                                                 
  vm-165-disk-0         vg-cluster01-storage01           -wi-ao----  27,00g                                                 
  vm-165-disk-1         vg-cluster01-storage01           -wi-ao----  81,00g                                                 
  vm-166-disk-0         vg-cluster01-storage01           -wi-a----- 202,00g                                                 
  vm-170-disk-0         vg-cluster01-storage01           -wi-a----- 102,00g                                                 
  vm-172-disk-0         vg-cluster01-storage01           -wi-a-----  41,00g                                                 
  vm-173-disk-0         vg-cluster01-storage01           -wi-a----- 201,00g                                                 
  vm-174-disk-0         vg-cluster01-storage01           -wi-ao---- 200,00g                                                 
  vm-175-disk-0         vg-cluster01-storage01           -wi-a-----  32,00g                                                 
  vm-176-disk-0         vg-cluster01-storage01           -wi-a----- 201,00g                                                 
  vm-177-disk-0         vg-cluster01-storage01           -wi-ao---- 102,00g                                                 
  vm-178-disk-0         vg-cluster01-storage01           -wi-ao---- 201,00g                                                 
  vm-180-disk-0         vg-cluster01-storage01           -wi-------  50,00g                                                 
  vm-184-disk-0         vg-cluster01-storage01           -wi-ao---- 305,00g                                                 
  vm-185-disk-0         vg-cluster01-storage01           -wi-ao----  20,00g                                                 
  vm-185-disk-1         vg-cluster01-storage01           -wi-ao----  50,00g                                                 
  vm-186-disk-0         vg-cluster01-storage01           -wi-ao----  20,00g                                                 
  vm-186-disk-1         vg-cluster01-storage01           -wi-ao----  50,00g                                                 
  vm-100-disk-0         vg-cluster01-storage01-ZYX -wi-a-----   1,00t                                                 
  vm-101-disk-0         vg-cluster01-storage01-ZYX -wi-ao----   1,00t                                                 
  vm-102-disk-0         vg-cluster01-storage01-ZYX -wi-a-----   1,00t                                                 
  vm-103-disk-0         vg-cluster01-storage01-ZYX -wi-a-----   1,00t                                                 
  vm-126-disk-0         vg-cluster01-storage01-ZYX -wi-a----- 200,00g                                                 
  lv-isotemplatestorage vg-isotemplatestorage            -wi-ao---- 190,00g                                                 
root@pm-01:~# pvs
  PV                                           VG                               Fmt  Attr PSize    PFree
  /dev/mapper/pm-cluster01-backup01            vg-cluster01-backup01            lvm2 a--  1024,00g 124,00g
  /dev/mapper/pm-cluster01-XYZ            vg-cluster01-XYZ            lvm2 a--     5,00t   3,00t
  /dev/mapper/pm-cluster01-demovolume          vg-cluster01-demovolume          lvm2 a--   200,00g 200,00g
  /dev/mapper/pm-cluster01-YXZ              vg-cluster01-YXZ              lvm2 a--     1,50t 536,00g
  /dev/mapper/pm-cluster01-online              vg-cluster01-online              lvm2 a--     1,50t 786,00g
  /dev/mapper/pm-cluster01-s4h                 vg-cluster01-s4h                 lvm2 a--     3,00t 222,00g
  /dev/mapper/pm-cluster01-storage01           vg-cluster01-storage01           lvm2 a--    20,00t   5,32t
  /dev/mapper/pm-cluster01-storage01-ZYX vg-cluster01-storage01-ZYX lvm2 a--     5,00t 824,00g
  /dev/mapper/pm-isotemplatestorage            vg-isotemplatestorage            lvm2 a--  1024,00g 834,00g
  /dev/sda3                                    pve                              lvm2 a--   135,62g  16,00g
root@pm-01:~# vgs
  VG                               #PV #LV #SN Attr   VSize    VFree
  pve                                1   3   0 wz--n-  135,62g  16,00g
  vg-cluster01-backup01              1   1   0 wz--n- 1024,00g 124,00g
  vg-cluster01-XYZ              1   2   0 wz--n-    5,00t   3,00t
  vg-cluster01-demovolume            1   0   0 wz--n-  200,00g 200,00g
  vg-cluster01-YXZ                1   2   0 wz--n-    1,50t 536,00g
  vg-cluster01-online                1   1   0 wz--n-    1,50t 786,00g
  vg-cluster01-s4h                   1   5   0 wz--n-    3,00t 222,00g
  vg-cluster01-storage01             1  78   0 wz--n-   20,00t   5,32t
  vg-cluster01-storage01-ZYX   1   5   0 wz--n-    5,00t 824,00g
  vg-isotemplatestorage              1   1   0 wz--n- 1024,00g 834,00g
root@pm-01:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0

My Storage is a Purestorage connected via FC with multipath.

It is working fine besides i have a stale device in /dev/mapper i can't get rid of.

root@pm-06:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0
/dev/mapper/vg--cluster01--s4h-vm--171--disk--1

root@pm-01:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0
 
Code: 
dmsetup info /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
dmsetup deps /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
What output do you get with the above commands? And where is the PV 'pm-*' located on (local/storage box)?
 
Code: 
root@pm-01:~# dmsetup info /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
Name:              vg--cluster01--storage01-vm--171--disk--0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 72
Number of targets: 1
UUID: LVM-uU4XEQ45hk00hlUS9tNVO2b2moEE5kDLYYrdhwxfaZa0EyhfIEx2k69HGGddVG2k

root@pm-01:~# dmsetup deps /dev/mapper/vg--cluster01--storage01-vm--171--disk--0
1 dependencies    : (253, 5)

root@pm-06:~# dmsetup info /dev/mapper/*171*
Name:              vg--cluster01--s4h-vm--171--disk--0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 93
Number of targets: 1
UUID: LVM-I0M2R2I3ibpyxr9CrynMAPO94E2HQTbBm1J06TfmX0GPlqf3U0ty2dzo6EQLNdrz

Name:              vg--cluster01--s4h-vm--171--disk--1
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 94
Number of targets: 1
UUID: LVM-I0M2R2I3ibpyxr9CrynMAPO94E2HQTbBjU8sacmed1ckfZOwhOrRlY2fpGlJDeCF

root@pm-06:~# dmsetup deps /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0: 1 dependencies   : (253, 1)
/dev/mapper/vg--cluster01--s4h-vm--171--disk--1: 1 dependencies   : (253, 1)

All /dev/mapper/pm* are on a storage box.
 
And where are the PV located on (local / storage box)? With 'lsblk' you could check on the node, if there is a device with matching 'major, minor' number.
 
All my pv used for virtual machines are on a SAN which is connected via Fiber Channel to the proxmox cluster.

First Cluster node (pm-01):
Code: 
sdg                                                        8:96   0    20T  0 disk
└─pm-cluster01-storage01                                 253:5    0    20T  0 mpath
  ├─vg--cluster01--storage01-vm--105--disk--0            253:7    0     1T  0 lvm
  ├─vg--cluster01--storage01-vm--105--disk--1            253:8    0   300G  0 lvm
  ├─vg--cluster01--storage01-vm--106--disk--0            253:9    0   510G  0 lvm
  ├─vg--cluster01--storage01-vm--107--disk--0            253:10   0   110G  0 lvm
  ├─vg--cluster01--storage01-vm--108--disk--0            253:11   0   110G  0 lvm
  ├─vg--cluster01--storage01-vm--109--disk--0            253:12   0   510G  0 lvm
[cut]
  ├─vg--cluster01--storage01-vm--168--disk--0            253:69   0   202G  0 lvm
  ├─vg--cluster01--storage01-vm--169--disk--0            253:70   0   202G  0 lvm
  ├─vg--cluster01--storage01-vm--170--disk--0            253:71   0   102G  0 lvm
  ├─vg--cluster01--storage01-vm--171--disk--0            253:72   0   202G  0 lvm
  ├─vg--cluster01--storage01-vm--172--disk--0            253:73   0    41G  0 lvm
  ├─vg--cluster01--storage01-vm--173--disk--0            253:74   0   201G  0 lvm
[cut]

root@pm-01:~# lvs|grep 171

root@pm-01:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--storage01-vm--171--disk--0

The VM with ID 171 does not exist anymore.

Sixth cluster node: (pm-06)
Code: 
sde                                                        8:64   0     3T  0 disk
└─pm-cluster01-s4h                                       253:1    0     3T  0 mpath
  ├─vg--cluster01--s4h-vm--183--disk--0                  253:88   0   2,1T  0 lvm
  ├─vg--cluster01--s4h-vm--183--disk--1                  253:89   0   100G  0 lvm
  ├─vg--cluster01--s4h-vm--183--disk--2                  253:90   0   100G  0 lvm
  ├─vg--cluster01--s4h-vm--179--disk--0                  253:91   0   500G  0 lvm
  ├─vg--cluster01--s4h-vm--179--disk--1                  253:92   0    50G  0 lvm
  ├─vg--cluster01--s4h-vm--171--disk--0                  253:93   0    20G  0 lvm
  └─vg--cluster01--s4h-vm--171--disk--1                  253:94   0    32G  0 lvm
root@pm-06:~# ls /dev/mapper/*171*
/dev/mapper/vg--cluster01--s4h-vm--171--disk--0 /dev/mapper/vg--cluster01--s4h-vm--171--disk--1
root@pm-06:~# lvs|grep 171
root@pm-06:~#

We do not have a VM with ID 171 on our cluster anymore. It got deleted (can't say when). When i tried to create a new VM on node 1 (pm-01) via WebUI it picked the id 171 automatically and tried to create the LV. It failed.

Code: 
 device-mapper: create ioctl on vg--cluster01--storage01-vm--171--disk--0LVM-uU4XEQ45hk00hlUS9tNVO2b2moEE5kDLengfrbdk3A8uDm9cbBbFpFQ5k4pgbfhH failed: Device or resource busy
TASK ERROR: unable to create VM 171 - error with cfs lock 'storage-vg-cluster01-storage01': lvcreate 'vg-cluster01-storage01/pve-vm-171' error: Failed to activate new LV.

I had a look around and i found /dev/mapper/vg--cluster01--storage01-vm--171--disk--0 on node 1.
And i found /dev/mapper/vg--cluster01--s4h-vm--171--disk--0 and /dev/mapper/vg--cluster01--s4h-vm--171--disk--1 on node 5/6 (these are two differnet volumes connected from the SAN).
Which do my understanding should not be possible since IDs in proxmox are uniq.

I am currently working around VM ID 171 simply by "touch 171.conf" in /etc/pve/nodes/pm-01/qemu-server/. Now Proxmox uses the next free ID.
 
Last edited:
Code: 
sdg                                                        8:96   0    20T  0 disk

└─pm-cluster01-storage01                                 253:5    0    20T  0 mpath
  ├─vg--cluster01--storage01-vm--171--disk--0            253:72   0   202G  0 lvm

Code: 
sde                                                        8:64   0     3T  0 disk
└─pm-cluster01-s4h                                       253:1    0     3T  0 mpath
  ├─vg--cluster01--s4h-vm--171--disk--0                  253:93   0    20G  0 lvm
  └─vg--cluster01--s4h-vm--171--disk--1                  253:94   0    32G  0 lvm
The VMID is unique, but the disk numbering starts at 0 on each storage.

Does a lvscan show them too? If the VM doesn't exist, then you could try to use the dmsetup to remove the mapping or reboot.
 
  • Like
Reactions: schoda
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom