[SOLVED] LVM (over iscsi) volume stuck and cannot be created

[gradinaruvasile]

We have a 3 node cluster and a SAN that is connected via iscsi (using lvm over iscsi).
I had to do some maintenance and move machines around and i noticed that one specific machine could not be migrated between nodes because of errors related to it's storage.

Initially it gave me some errors when i wanted to migrate the machine to another node, along the lines of

device-mapper: create ioctl on san_lvm_volgroup-vm--135--disk--0 LVM-6kXN8w5HcxnsGojMrnMBI1IaPJ0kS3YSQRpboe0uwleV3UNQq8mJOlkrSDupKSi3 failed: Device or resource busy

I migrated the disk to another storage (one cluster member's nfs shared disk), it went ok.
Now i wanted to migrate it back to the SAN and i still get errors like this:

create full clone of drive scsi0 (srv005:135/vm-135-disk-0.qcow2)
  device-mapper: create ioctl on san_lvm_volgroup-vm--135--disk--0 LVM-6kXN8w5HcxnsGojMrnMBI1IaPJ0kS3YSQRpboe0uwleV3UNQq8mJOlkrSDupKSi3 failed: Device or resource busy
TASK ERROR: storage migration failed: error with cfs lock 'storage-zesan-lvm': lvcreate 'san_lvm_volgroup/vm-135-disk-0' error:   Failed to activate new LV san_lvm_volgroup/vm-135-disk-0.

lvscan and lvdisplay does not show "vm-135-disk-0". I suppose something is stuck somewhere but i have no idea. I migrated other machines' disks to and from the SAN LVM succesfully.

Where should i start debugging this?

 

[Dominic]

Hi!

Have you been able to solve your problem already? If not, what version of Proxmox VE are you using pveversion -v?

 

[gradinaruvasile]

Hi,
Still get the same error. I moved the drive to another storage for now. The issue seems to be related to iscsi having something stuck somewhere.

Pveversion (we use the enterprise repos):

# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.13-3-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-3
pve-kernel-helper: 6.1-3
pve-kernel-5.0: 6.0-11
pve-kernel-4.15: 5.4-7
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-11
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-4
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[gradinaruvasile]

I managed to move the VM to another node, move the disk to the SAN. Then i could move the VM to a third node.
But i still get this error when i try to move it back to the original node i encountered the error on.
So that server is the only one with these issues.

 

[Dominic]

Please check out if dmsetup can help you, dmsetup table would be a start, for example. Two possibly related threads: 1 & 2.

 

[gradinaruvasile]

Thanks.
This worked.

Edit - some details:
Basically i identified the stuck device with

dmsetup table |grep 135

Then removed it with

dmsetup remove san_lvm_volgroup-vm--135--disk--0

 

[Denny Fuchs]

hi,

I have this issue also with PVE7 on two different PVE7 clusters, which has ISCSI as shared device. I know this workaround with dmesetup remove .. but I do not really understand, the source of this issue. I can be only a race condition, as it happens not always .. but pretty often for migrate or create new VMs.

cu denny

 

[fweber]

Hi, could it be the case that a VM with the same VMID existed previously and was destroyed, and then the PVE node (on which VM creation now fails with "Device or resource busy", or which is the target of a migration that now fails) was rebooted? If yes, the root cause might be LVM autoactivation on boot, there is also an open bug report with more details [1] and I'm currently working on a fix.

If the shared LVM VG in question contains only PVE-managed VM disks, you could try whether disabling LVM autoactivation for that VG fixes the issue:

vgchange VG_NAME --setautoactivation n

For this to take effect, you'd need to reboot all nodes.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=4997

 

[Denny Fuchs]

hi @fweber

don't think so. It does not happen anymore, if you add a filter in lvm.conf like this:

     global_filter=["r|/dev/zd.*|","r|/dev/mapper/.*-vm--[0-9]+--disk--[0-9]+|"]

Most of the issues we had, for create new VMs where no ID was reused.

cu denny

 

[fweber]

don't think so. It does not happen anymore, if you add a filter in lvm.conf like this:

     global_filter=["r|/dev/zd.*|","r|/dev/mapper/.*-vm--[0-9]+--disk--[0-9]+|"]

Most of the issues we had, for create new VMs where no ID was reused.

Thank you for the information. I suppose the "r|/dev/mapper/.*-vm--[0-9]+--disk--[0-9]+|" filter did the trick for you then? I wonder why it is necessary -- if its purpose is to prevent LVM from recognizing LVs as PVs, the scan_lvs=0 setting should already avoid that. Could you please attach /etc/lvm/lvm.conf and the output of the following commands?

lvmconfig --typeconfig full devices/scan_lvs
ls -al /dev/mapper/* # feel free to censor snapshot names here, if needed

 

[Denny Fuchs]

hi @fweber

gladly .. I wanted to create a new developer VM and in this case .. exactly this happens, what you told about your issue. An existing device will be reused. Maybe .. you can still use this info:

root@fra-corp-pmox-03:[~]: pveversion 
pve-manager/7.4-3/9002ab8a (running kernel: 5.15.104-1-pve)

root@fra-corp-pmox-03:[~]: multipath -ll
mpathc (36d039ea00007e79d000002165e74d6d5) dm-0 LENOVO,DE_Series
size=6.0T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 15:0:0:1 sdc 8:32 active ready running
| `- 18:0:0:1 sdf 8:80 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 16:0:0:1 sde 8:64 active ready running
  `- 17:0:0:1 sdd 8:48 active ready running
root@fra-corp-pmox-03:[~]:

 device-mapper: create ioctl on lenovo--san-vm--139--disk--0 LVM-qMITWfpvcu12biHgSn1VYXvbKHlByZjZf8ZGwYBEzYsYlQ0FAUu3eLXdjtVoGP6Y failed: Device or resource busy
TASK ERROR: unable to create VM 139 - lvcreate 'lenovo-san/vm-139-disk-0' error:   Failed to activate new LV lenovo-san/vm-139-disk-0.

root@fra-corp-pmox-03:[~]: lvmconfig --typeconfig full devices/scan_lvs
scan_lvs=0

* LVM.conf

config {
    checks = 1
    abort_on_errors = 0
    profile_dir = "/etc/lvm/profile"
}
devices {
    dir = "/dev"
    scan = [ "/dev" ]
    obtain_device_list_from_udev = 1
    external_device_info_source = "none"
    sysfs_scan = 1
    scan_lvs = 0
    multipath_component_detection = 1
    md_component_detection = 1
    fw_raid_component_detection = 0
    md_chunk_alignment = 1
    data_alignment_detection = 1
    data_alignment = 0
    data_alignment_offset_detection = 1
    ignore_suspended_devices = 0
    ignore_lvm_mirrors = 1
    require_restorefile_with_uuid = 1
    pv_min_size = 2048
    issue_discards = 0
    allow_changes_with_duplicate_pvs = 0
    allow_mixed_block_sizes = 0
}
allocation {
    maximise_cling = 1
    use_blkid_wiping = 1
    wipe_signatures_when_zeroing_new_lvs = 1
    mirror_logs_require_separate_pvs = 0
}
log {
    verbose = 0
    silent = 0
    syslog = 1
    overwrite = 0
    level = 0
    command_names = 0
    prefix = "  "
    activation = 0
    debug_classes = [ "memory", "devices", "io", "activation", "allocation", "metadata", "cache", "locking", "lvmpolld", "dbus" ]
}
backup {
    backup = 1
    backup_dir = "/etc/lvm/backup"
    archive = 1
    archive_dir = "/etc/lvm/archive"
    retain_min = 10
    retain_days = 30
}
shell {
    history_size = 100
}
global {
    umask = 077
    test = 0
    units = "r"
    si_unit_consistency = 1
    suffix = 1
    activation = 1
    proc = "/proc"
    etc = "/etc"
    wait_for_locks = 1
    locking_dir = "/run/lock/lvm"
    prioritise_write_locks = 1
    abort_on_internal_errors = 0
    metadata_read_only = 0
    mirror_segtype_default = "raid1"
    raid10_segtype_default = "raid10"
    sparse_segtype_default = "thin"
    use_lvmlockd = 0
    system_id_source = "none"
    use_lvmpolld = 1
    notify_dbus = 1
}
activation {
    checks = 0
    udev_sync = 1
    udev_rules = 1
    retry_deactivation = 1
    missing_stripe_filler = "error"
    raid_region_size = 2048
    raid_fault_policy = "warn"
    mirror_image_fault_policy = "remove"
    mirror_log_fault_policy = "allocate"
    snapshot_autoextend_threshold = 100
    snapshot_autoextend_percent = 20
    thin_pool_autoextend_threshold = 100
    thin_pool_autoextend_percent = 20
    monitoring = 1
    activation_mode = "degraded"
}
dmeventd {
}
devices {
     global_filter=["r|/dev/zd.*|"]
 }

lrwxrwxrwx  1 root root       8 Jul 25 14:31 lenovo--san-vm--139--disk--0 -> ../dm-10
root@fra-corp-pmox-03:[~]: fdisk -l /dev/mapper/lenovo--san-vm--139--disk--0 
Disk /dev/mapper/lenovo--san-vm--139--disk--0: 35 GiB, 37580963840 bytes, 73400320 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x8d3980ec

Device                                         Boot   Start      End  Sectors  Size Id Type
/dev/mapper/lenovo--san-vm--139--disk--0-part1         2048  1026047  1024000  500M 83 Linux
/dev/mapper/lenovo--san-vm--139--disk--0-part2      1026048 73400319 72374272 34.5G 8e Linux LVM

and the best

root@fra-corp-pmox-03:[~]: ls  -a /dev/mapper/*
corrupted double-linked list
Aborted

but .. after a few mins (while collecting other infos) ..

root@fra-corp-pmox-03:[~]: ls -la /dev/mapper/*
crw------- 1 root root 10, 236 Jul 25 14:31 /dev/mapper/control
lrwxrwxrwx 1 root root       8 Jul 26 08:54 /dev/mapper/lenovo--san-vm--100--disk--0 -> ../dm-72
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--101--disk--0 -> ../dm-30
lrwxrwxrwx 1 root root       8 Aug 23 15:28 /dev/mapper/lenovo--san-vm--102--disk--0 -> ../dm-17
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--103--disk--0 -> ../dm-27
lrwxrwxrwx 1 root root       8 Jul 25 14:33 /dev/mapper/lenovo--san-vm--104--disk--0 -> ../dm-20
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--105--disk--0 -> ../dm-35
lrwxrwxrwx 1 root root       8 Jan 17 06:00 /dev/mapper/lenovo--san-vm--106--disk--0 -> ../dm-31
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--108--disk--0 -> ../dm-23
lrwxrwxrwx 1 root root       8 Jul 25 14:34 /dev/mapper/lenovo--san-vm--109--disk--0 -> ../dm-66
lrwxrwxrwx 1 root root       8 Jul 25 14:34 /dev/mapper/lenovo--san-vm--109--disk--1 -> ../dm-67
lrwxrwxrwx 1 root root       8 Sep 19 17:35 /dev/mapper/lenovo--san-vm--111--disk--0 -> ../dm-13
lrwxrwxrwx 1 root root       8 Jan 17 06:00 /dev/mapper/lenovo--san-vm--113--disk--0 -> ../dm-24
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--114--disk--0 -> ../dm-73
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--114--disk--1 -> ../dm-74
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--115--disk--0 -> ../dm-56
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--115--disk--1 -> ../dm-57
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--117--disk--0 -> ../dm-26
lrwxrwxrwx 1 root root       8 Jul 25 14:36 /dev/mapper/lenovo--san-vm--118--disk--0 -> ../dm-91
lrwxrwxrwx 1 root root       8 Jul 25 14:36 /dev/mapper/lenovo--san-vm--118--disk--1 -> ../dm-92
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--119--disk--0 -> ../dm-16
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--119--disk--1 -> ../dm-18
lrwxrwxrwx 1 root root       8 Jul 25 16:47 /dev/mapper/lenovo--san-vm--120--disk--0 -> ../dm-12
lrwxrwxrwx 1 root root       8 Jul 25 16:47 /dev/mapper/lenovo--san-vm--120--disk--1 -> ../dm-19
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--122--disk--0 -> ../dm-32
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--122--disk--1 -> ../dm-33
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--123--disk--0 -> ../dm-14
lrwxrwxrwx 1 root root       7 Jan 17 06:01 /dev/mapper/lenovo--san-vm--124--disk--0 -> ../dm-7
lrwxrwxrwx 1 root root       8 Jul 25 14:38 /dev/mapper/lenovo--san-vm--125--disk--0 -> ../dm-36
lrwxrwxrwx 1 root root       8 Jul 25 14:38 /dev/mapper/lenovo--san-vm--125--disk--1 -> ../dm-45
lrwxrwxrwx 1 root root       7 Jul 25 14:31 /dev/mapper/lenovo--san-vm--126--disk--0 -> ../dm-2
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--127--disk--0 -> ../dm-28
lrwxrwxrwx 1 root root       7 Jul 25 14:31 /dev/mapper/lenovo--san-vm--129--disk--0 -> ../dm-6
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--129--disk--1 -> ../dm-65
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--130--disk--0 -> ../dm-54
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--130--disk--3 -> ../dm-62
lrwxrwxrwx 1 root root       8 Aug 23 15:28 /dev/mapper/lenovo--san-vm--131--disk--0 -> ../dm-37
lrwxrwxrwx 1 root root       8 Aug 23 15:28 /dev/mapper/lenovo--san-vm--131--disk--1 -> ../dm-46
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--132--disk--1 -> ../dm-58
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--132--disk--2 -> ../dm-59
lrwxrwxrwx 1 root root       8 Jan 17 06:01 /dev/mapper/lenovo--san-vm--133--disk--0 -> ../dm-34
lrwxrwxrwx 1 root root       7 Jan 17 06:01 /dev/mapper/lenovo--san-vm--134--disk--0 -> ../dm-3
lrwxrwxrwx 1 root root       8 Jan 17 06:01 /dev/mapper/lenovo--san-vm--135--disk--0 -> ../dm-15
lrwxrwxrwx 1 root root       8 Jan 17 06:02 /dev/mapper/lenovo--san-vm--137--disk--0 -> ../dm-29
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--138--disk--0 -> ../dm-11
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--139--disk--0 -> ../dm-10
lrwxrwxrwx 1 root root       7 Sep  6 11:23 /dev/mapper/lenovo--san-vm--142--disk--0 -> ../dm-4
lrwxrwxrwx 1 root root       7 Sep  6 11:23 /dev/mapper/lenovo--san-vm--142--disk--1 -> ../dm-5
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--143--disk--0 -> ../dm-25
lrwxrwxrwx 1 root root       8 Jul 25 14:41 /dev/mapper/lenovo--san-vm--144--disk--0 -> ../dm-39
lrwxrwxrwx 1 root root       8 Jul 25 14:41 /dev/mapper/lenovo--san-vm--144--disk--1 -> ../dm-48
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--145--disk--0 -> ../dm-40
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--145--disk--1 -> ../dm-49
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--146--disk--0 -> ../dm-41
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--146--disk--1 -> ../dm-50
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--147--disk--0 -> ../dm-42
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--147--disk--1 -> ../dm-51
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--148--disk--0 -> ../dm-43
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--148--disk--1 -> ../dm-52
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--149--disk--0 -> ../dm-44
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--149--disk--1 -> ../dm-53
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--150--disk--0 -> ../dm-55
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--151--disk--0 -> ../dm-99
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--151--disk--1 -> ../dm-100
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--152--disk--0 -> ../dm-93
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--152--disk--1 -> ../dm-94
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--153--disk--0 -> ../dm-60
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--153--disk--1 -> ../dm-61
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--154--disk--0 -> ../dm-70
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--154--disk--1 -> ../dm-71
lrwxrwxrwx 1 root root       8 Aug 23 15:26 /dev/mapper/lenovo--san-vm--155--disk--0 -> ../dm-68
lrwxrwxrwx 1 root root       8 Aug 23 15:26 /dev/mapper/lenovo--san-vm--155--disk--1 -> ../dm-69
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--156--disk--0 -> ../dm-75
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--156--disk--1 -> ../dm-76
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--157--disk--0 -> ../dm-95
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--157--disk--1 -> ../dm-96
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--158--disk--0 -> ../dm-77
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--158--disk--1 -> ../dm-78
lrwxrwxrwx 1 root root       8 Jul 25 16:47 /dev/mapper/lenovo--san-vm--159--disk--0 -> ../dm-79
lrwxrwxrwx 1 root root       8 Jul 25 16:47 /dev/mapper/lenovo--san-vm--159--disk--1 -> ../dm-80
lrwxrwxrwx 1 root root       8 Aug 23 15:29 /dev/mapper/lenovo--san-vm--160--disk--0 -> ../dm-81
lrwxrwxrwx 1 root root       8 Aug 23 15:29 /dev/mapper/lenovo--san-vm--160--disk--1 -> ../dm-82
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--161--disk--0 -> ../dm-83
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--161--disk--1 -> ../dm-84
lrwxrwxrwx 1 root root       8 Aug 23 15:29 /dev/mapper/lenovo--san-vm--162--disk--0 -> ../dm-85
lrwxrwxrwx 1 root root       8 Aug 23 15:29 /dev/mapper/lenovo--san-vm--162--disk--1 -> ../dm-86
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--163--disk--0 -> ../dm-87
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--163--disk--1 -> ../dm-88
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--164--disk--0 -> ../dm-89
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--164--disk--1 -> ../dm-90
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--165--disk--0 -> ../dm-97
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--165--disk--1 -> ../dm-98
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--166--disk--0 -> ../dm-101
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--166--disk--1 -> ../dm-102
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--167--disk--0 -> ../dm-103
lrwxrwxrwx 1 root root       9 Aug 23 15:30 /dev/mapper/lenovo--san-vm--168--disk--0 -> ../dm-104
lrwxrwxrwx 1 root root       9 Aug 23 15:30 /dev/mapper/lenovo--san-vm--168--disk--1 -> ../dm-107
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--169--disk--0 -> ../dm-105
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--169--disk--1 -> ../dm-108
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--170--disk--0 -> ../dm-106
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--170--disk--1 -> ../dm-109
lrwxrwxrwx 1 root root       9 Aug 23 15:26 /dev/mapper/lenovo--san-vm--171--disk--0 -> ../dm-110
lrwxrwxrwx 1 root root       9 Aug 23 15:26 /dev/mapper/lenovo--san-vm--171--disk--1 -> ../dm-111
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--172--disk--0 -> ../dm-112
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--172--disk--1 -> ../dm-113
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--173--disk--0 -> ../dm-114
lrwxrwxrwx 1 root root       9 Jul 25 14:31 /dev/mapper/lenovo--san-vm--173--disk--1 -> ../dm-115
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--600--disk--0 -> ../dm-63
lrwxrwxrwx 1 root root       8 Jul 25 14:31 /dev/mapper/lenovo--san-vm--600--disk--1 -> ../dm-64
lrwxrwxrwx 1 root root       7 Jul 25 14:31 /dev/mapper/mpathc -> ../dm-0
lrwxrwxrwx 1 root root       7 Jan 17 10:27 /dev/mapper/mpathc-part1 -> ../dm-1

so, what I'm doing now .. is:

root@fra-corp-pmox-03:[~]: dmsetup remove lenovo--san-vm--139--disk--0

And recreate the VM .. which will then work

  Logical volume "vm-139-disk-0" created.
scsi0: successfully created disk 'lenovo-san-01:vm-139-disk-0,discard=on,iothread=1,size=25G,ssd=1'
  Logical volume "vm-139-disk-1" created.
scsi1: successfully created disk 'lenovo-san-01:vm-139-disk-1,discard=on,iothread=1,size=32G,ssd=1'
TASK OK
[CODE]

 

[fweber]

hi @fweber

gladly .. I wanted to create a new developer VM and in this case .. exactly this happens, what you told about your issue. An existing device will be reused.

I see, thanks for checking! But in the LVM config you sent, it seems like the global_filter does not actually include "r|/dev/mapper/.*-vm--[0-9]+--disk--[0-9]+|"?

and the best

root@fra-corp-pmox-03:[~]: ls  -a /dev/mapper/*
corrupted double-linked list
Aborted

ls shouldn't produce an error like this -- do you see such errors more frequently? Maybe this could hint at faulty RAM, might make sense to run a memtest to rule this out.