Clone creation fails -

Aug 3, 2022
2
0
1
Hi,

I do have 3-node cluster with latest up-to-date version of Proxmox VE 7.2-7. This is fresh installation. We used to have PVE5.x

Technical Information:
Storage back-end configuration : SAN (iSCSI+LVM)
lvm.conf global_filter value:
global_filter = [ "r|/dev/zd.*|", "r|/dev/mapper/pve-.*|", "r|/dev/mapper/small--pool.*|", "r|/dev/mapper/medium--pool.*|", "r|/dev/mapper/large--pool.*|", "r|/dev/mapper/template--pool.*|" ]
I am bypassing the scan of the SAN-Based Volumes

I do have OS template from which we clone (Full clone mode) VMs. THis is done automatically with API calls.

For some reason the clone creation fails. Here is the example of the error from the TASK:
"create full clone of drive scsi0 (pve-template-pool-datastore:vm-106-disk-0)
device-mapper: create ioctl on small--pool2-vm--159--disk--0 LVM-bTzHDpropJH4Td1wQwNT5oskPWvcNATTsn1InnBnH1qZv1edqGzlxnj1qMWW0BB3 failed: Device or resource busy
TASK ERROR: clone failed: lvcreate 'small-pool2/vm-159-disk-0' error: Failed to activate new LV small-pool2/vm-159-disk-0."

In this case the corrupt VM iD is 159.

========================================================================================================
Here are the SYSLOGs:
Aug 02 07:55:07 pve-002 pvedaemon[544590]: <root@pam> successful auth for user 'jenkins-prox@stratinfotech.com'
Aug 02 07:55:07 pve-002 pveproxy[1253665]: Clearing outdated entries from certificate cache
Aug 02 07:55:17 pve-002 pvedaemon[616446]: jenkins-prox@stratinfotech.com starting task UPID:pve-002:0014317B:0333B024:62E910A5:qmclone:106:jenkins-prox@stratinfotech.com:
Aug 02 07:55:19 pve-002 pvedaemon[1323387]: VM 106 qmp command failed - VM 106 not running
Aug 02 07:55:20 pve-002 pvedaemon[1323387]: clone failed: lvcreate 'small-pool1/vm-115-disk-0' error: Failed to activate new LV small-pool1/vm-115-disk-0.
Aug 02 07:55:20 pve-002 pvedaemon[616446]: jenkins-prox@stratinfotech.com end task UPID:pve-002:0014317B:0333B024:62E910A5:qmclone:106:jenkins-prox@stratinfotech.com: clone failed: lvcreate 'small-pool1/vm-115-disk-0' error: Failed to activate new LV small-pool1/vm-115-disk-0.
Aug 02 07:57:25 pve-002 pvedaemon[616446]: <root@pam> successful auth for user 'jenkins-prox@stratinfotech.com'
Aug 02 07:57:28 pve-002 pveproxy[1253666]: Clearing outdated entries from certificate cache
Aug 02 07:57:34 pve-002 pvedaemon[556778]: jenkins-prox@stratinfotech.com starting task UPID:pve-002:00143302:0333E5D0:62E9112E:qmclone:106:jenkins-prox@stratinfotech.com:
Aug 02 07:57:36 pve-002 pvedaemon[1323778]: VM 106 qmp command failed - VM 106 not running
Aug 02 07:57:37 pve-002 pvedaemon[1323778]: clone failed: lvcreate 'small-pool1/vm-115-disk-0' error: Failed to activate new LV small-pool1/vm-115-disk-0.
Aug 02 07:57:37 pve-002 pvedaemon[556778]: jenkins-prox@stratinfotech.com end task UPID:pve-002:00143302:0333E5D0:62E9112E:qmclone:106:jenkins-prox@stratinfotech.com: clone failed: lvcreate 'small-pool1/vm-115-disk-0' error: Failed to activate new LV small-pool1/vm-115-disk-0.
Aug 02 08:17:01 pve-002 CRON[1326621]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 02 08:17:01 pve-002 CRON[1326622]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 02 08:17:01 pve-002 CRON[1326621]: pam_unix(cron:session): session closed for user root
Aug 02 08:42:40 pve-002 pmxcfs[1504]: [dcdb] notice: data verification successful
Aug 02 09:17:01 pve-002 CRON[1335351]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Aug 02 09:17:01 pve-002 CRON[1335352]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 02 09:17:01 pve-002 CRON[1335351]: pam_unix(cron:session): session closed for user root
Aug 02 09:28:57 pve-002 pvedaemon[616446]: <root@pam> successful auth for user 'andyb@stratinfotech.com'
Aug 02 09:30:25 pve-002 pvedaemon[544590]: andyb@stratinfotech.com starting task UPID:pve-002:00146800:033C65D1:62E926F1:qmclone:106:andyb@stratinfotech.com:
Aug 02 09:30:27 pve-002 pvedaemon[1337344]: VM 106 qmp command failed - VM 106 not running
Aug 02 09:30:28 pve-002 pvedaemon[1337344]: clone failed: lvcreate 'small-pool1/vm-115-disk-0' error: Failed to activate new LV small-pool1/vm-115-disk-0.
Aug 02 09:30:28 pve-002 pvedaemon[544590]: andyb@stratinfotech.com end task UPID:pve-002:00146800:033C65D1:62E926F1:qmclone:106:andyb@stratinfotech.com: clone failed: lvcreate 'small-pool1/vm-115-disk-0' error: Failed to activate new LV small-pool1/vm-115-disk-0.
Aug 02 09:33:24 pve-002 pvedaemon[616446]: andyb@stratinfotech.com starting task UPID:pve-002:001469D7:033CABC7:62E927A4:qmclone:106:andyb@stratinfotech.com:


========================================================================================================
As you can see in the example above the faulty or corrupt VM ID is 115.

Possible reasons:
1. My global_filter that I pasted above was just working fine with my PROXMOX cluster 5.x. Is there something missing ?
2. Our automated (Jenkins) system that creates and deletes VMs has not changed (pvesh delete /nodes/{node}/qemu/{vmid}). I know that the new PVE 7.2.x has new switches when it comes to the VM deletion:
"Purge from job configurations"
"Destroy unreferenced disks owned by guest"
Is it necessary to add those checks into the API command line (Jenkins call) while deleting the VM ? is the absense of those switches the root cause ?
My temporary workaround: I am forced to clone the VM manually through the GUI to reserve the faulty VM ID (after severals fails I am able to generate VM manually off the template) and then Jenkins users are able to spin up other VMs with other nextID.

I would appreciate your prompt response.

Regards,
Andy
 
First - this issue is most definitely not caused by any API change at PVE level or how you issue your API request.

Its possible that it may be related to your LVM filter, or rather how it is processed today. The error happens below PVE application. Essentially at the end of your request PVE executes standard LVM activate command. That LVM activation fails. It will likely fail if you try to do it manually using "lvchange". If the resource is busy that means something gets a hold of it, perhaps you need to expand your filter - device naming have changed with kernel updates.

Quite a lot has changed between PVE5 and PVE7: the underlying OS, most toolsets (including LVM) and of course PVE itself. More bumpers were put in place at every level.
You said you are using SAN/LVM combination - is this used as shared storage? If it is - is LVM deployed in a supported way, ie Thin? I've never worked with PVE5 so as a wild guess - it may have allowed SAN/LVM usage in a way that 7 does not.

You can try to get more information by increasing debug verbosity of the LVM: https://www.thegeekdiary.com/centos-rhel-how-to-log-all-lvm-commands/

Beyond that, if this is a production issue and you being a paying customer - I recommend opening a proper ticket with PVE support.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
yeah, definitely more information required here.. e.g.,
  • pveversion -v
  • storage.cfg
  • lvm config
  • pvs / vgs / lvs output
  • any warnings/errors from journal relating to LVM or iSCSI
 
Thank you bbgreek17,

Yes the storage is shared amongst 3 nodes in the cluster. According to Proxmox documentation/matrix (Table 1. Available storage types section) in https://pve.proxmox.com/wiki/Storage it is specified that table LVM+Shared = No2. However there is very important asterisk:
"2: It is possible to use LVM on top of an iSCSI or FC-based storage. That way you get a shared LVM storage."

Hello Fabian,
Here is the requested information:

root@pve-002:/# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.4-1
proxmox-backup-file-restore: 2.2.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

root@pve-002:/etc/pve# cat storage.cfg
dir: local
path /var/lib/vz
content vztmpl,backup,iso

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

iscsi: pve-template-pool
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:template-pool-v1ebd051040586e9b.0000004a.5ee56b8b
content none

lvm: pve-template-pool-datastore
vgname template-pool
base pve-template-pool:0.0.0.scsi-29c99e309b826d2826c9ce9008b6be55e
content rootdir,images
shared 1

iscsi: pve-small-pool1
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:small-pool1-v1ebd051040586e9b.00000043.5ee56b8b
content none

iscsi: pve-small-pool2
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:small-pool2-v1ebd051040586e9b.00000044.5ee56b8b
content none

iscsi: pve-small-pool3
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:small-pool3-v1ebd051040586e9b.00000045.5ee56b8b
content none

iscsi: pve-small-pool4
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:small-pool4-v1ebd051040586e9b.00000046.5ee56b8b
content none

iscsi: pve-small-pool5
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:small-pool5-v1ebd051040586e9b.00000047.5ee56b8b
content none

iscsi: pve-medium-pool1
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:medium-pool1-v1ebd051040586e9b.00000041.5ee56b8b
content none

iscsi: pve-medium-pool2
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:medium-pool2-v1ebd051040586e9b.00000042.5ee56b8b
content none

iscsi: pve-large-pool1
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:large-pool1-v1ebd051040586e9b.00000048.5ee56b8b
content none

iscsi: pve-large-pool2
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:large-pool2-v1ebd051040586e9b.00000049.5ee56b8b
content none

lvm: pve-large-pool1-datastore
vgname large-pool1
base pve-large-pool1:0.0.0.scsi-23e70a1859cfe97026c9ce9008b6be55e
content images,rootdir
shared 1

lvm: pve-large-pool2-datastore
vgname large-pool2
base pve-large-pool2:0.0.0.scsi-20fc516b8f31a4b626c9ce9008b6be55e
content rootdir,images
shared 1

lvm: pve-medium-pool1-datastore
vgname medium-pool1
base pve-medium-pool1:0.0.0.scsi-297c9037ede9b144c6c9ce9008b6be55e
content rootdir,images
shared 1

lvm: pve-medium-pool2-datastore
vgname medium-pool2
base pve-medium-pool2:0.0.0.scsi-224e606caa2a7cd656c9ce9008b6be55e
content rootdir,images
shared 1

lvm: pve-small-pool1-datastore
vgname small-pool1
base pve-small-pool1:0.0.0.scsi-2f5d0cc964db3f6d66c9ce9008b6be55e
content images,rootdir
shared 1

lvm: pve-small-pool2-datastore
vgname small-pool2
base pve-small-pool2:0.0.0.scsi-20a48ba7806406f2d6c9ce9008b6be55e
content images,rootdir
shared 1

lvm: pve-small-pool3-datastore
vgname small-pool3
base pve-small-pool3:0.0.0.scsi-2db4632db535c2c3a6c9ce9008b6be55e
content rootdir,images
shared 1

lvm: pve-small-pool4-datastore
vgname small-pool4
base pve-small-pool4:0.0.0.scsi-26871fcfc2168a9ad6c9ce9008b6be55e
content images,rootdir
shared 1

lvm: pve-small-pool5-datastore
vgname small-pool5
base pve-small-pool5:0.0.0.scsi-239a0c1f9c273dacb6c9ce9008b6be55e
content images,rootdir
shared 1

iscsi: pve-backups
disable
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:pve-backups-v1ebd051040586e9b.0000004b.5ee56b8b
content none

dir: Backups
disable
path /backups
content backup
prune-backups keep-all=1
shared 0

iscsi: pve-large-pool3
portal 10.0.0.47
target iqn.2007-11.com.nimblestorage:large-pool3-v1ebd051040586e9b.0000004d.5ee56b8b
content none

lvm: pve-large-pool3-datastore
vgname large-pool3
base pve-large-pool3:0.0.0.scsi-253fa0e1b69db30576c9ce9008b6be55e
content rootdir,images
shared 1


I have also attached lvm.conf. Global Filter has no Duplicate Warnings....

root@pve-002:/# pvs
PV VG Fmt Attr PSize PFree
/dev/sdb3 pve lvm2 a-- 117.25g 14.62g
/dev/sdc small-pool5 lvm2 a-- <2.00t 1.33t
/dev/sdd large-pool1 lvm2 a-- <4.00t 1.24t
/dev/sde medium-pool2 lvm2 a-- <3.00t <1.98t
/dev/sdg small-pool1 lvm2 a-- <2.00t <1.31t
/dev/sdh large-pool2 lvm2 a-- <4.00t 1.24t
/dev/sdi small-pool2 lvm2 a-- <2.00t <1.37t
/dev/sdj large-pool3 lvm2 a-- <4.00t 3.31t
/dev/sdk template-pool lvm2 a-- <800.00g <249.00g
/dev/sdl small-pool3 lvm2 a-- <2.00t <1.37t
/dev/sdm small-pool4 lvm2 a-- <2.00t <1.33t
/dev/sdn medium-pool1 lvm2 a-- <3.00t <2.33t

root@pve-002:/# vgs
VG #PV #LV #SN Attr VSize VFree
large-pool1 1 4 0 wz--n- <4.00t 1.24t
large-pool2 1 4 0 wz--n- <4.00t 1.24t
large-pool3 1 1 0 wz--n- <4.00t 3.31t
medium-pool1 1 5 0 wz--n- <3.00t <2.33t
medium-pool2 1 8 0 wz--n- <3.00t <1.98t
pve 1 3 0 wz--n- 117.25g 14.62g
small-pool1 1 12 0 wz--n- <2.00t <1.31t
small-pool2 1 11 0 wz--n- <2.00t <1.37t
small-pool3 1 11 0 wz--n- <2.00t <1.37t
small-pool4 1 11 0 wz--n- <2.00t <1.33t
small-pool5 1 12 0 wz--n- <2.00t 1.33t
template-pool 1 6 0 wz--n- <800.00g <249.00g
root@pve-002:/# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
vm-123-disk-0 large-pool1 -wi-a----- 706.00g
vm-134-disk-0 large-pool1 -wi-a----- 706.00g
vm-142-disk-0 large-pool1 -wi-a----- 706.00g
vm-160-disk-0 large-pool1 -wi-a----- 706.00g
vm-127-disk-0 large-pool2 -wi-a----- 706.00g
vm-137-disk-0 large-pool2 -wi-a----- 706.00g
vm-139-disk-0 large-pool2 -wi-a----- 706.00g
vm-151-disk-0 large-pool2 -wi-a----- 706.00g
vm-148-disk-0 large-pool3 -wi-ao---- 706.00g
vm-131-disk-0 medium-pool1 -wi-a----- 145.00g
vm-147-disk-0 medium-pool1 -wi-a----- 132.00g
vm-149-disk-0 medium-pool1 -wi-a----- 145.00g
vm-164-disk-0 medium-pool1 -wi-a----- 133.00g
vm-172-disk-0 medium-pool1 -wi-a----- 133.00g
vm-116-disk-0 medium-pool2 -wi-a----- 133.00g
vm-121-disk-0 medium-pool2 -wi-ao---- 132.00g
vm-132-disk-0 medium-pool2 -wi-a----- 106.00g
vm-146-disk-0 medium-pool2 -wi-a----- 145.00g
vm-153-disk-0 medium-pool2 -wi-a----- 133.00g
vm-165-disk-0 medium-pool2 -wi-a----- 132.00g
vm-176-disk-0 medium-pool2 -wi-a----- 133.00g
vm-188-disk-0 medium-pool2 -wi------- 133.00g
data pve twi-a-tz-- <39.38g 0.00 1.59
root pve -wi-ao---- 29.25g
swap pve -wi-ao---- 32.00g
vm-108-disk-0 small-pool1 -wi-a----- 57.00g
vm-114-disk-0 small-pool1 -wi-ao---- 57.00g
vm-120-disk-0 small-pool1 -wi-a----- 57.00g
vm-124-disk-0 small-pool1 -wi-ao---- 57.00g
vm-129-disk-0 small-pool1 -wi-a----- 80.00g
vm-138-disk-0 small-pool1 -wi-ao---- 57.00g
vm-143-disk-0 small-pool1 -wi------- 57.00g
vm-152-disk-0 small-pool1 -wi------- 57.00g
vm-159-disk-0 small-pool1 -wi-a----- 57.00g
vm-169-disk-0 small-pool1 -wi-a----- 57.00g
vm-177-disk-0 small-pool1 -wi------- 57.00g
vm-179-disk-0 small-pool1 -wi-a----- 57.00g
vm-105-disk-0 small-pool2 -wi-a----- 57.00g
vm-110-disk-0 small-pool2 -wi-ao---- 57.00g
vm-111-disk-0 small-pool2 -wi-a----- 57.00g
vm-113-disk-0 small-pool2 -wi-a----- 77.00g
vm-126-disk-0 small-pool2 -wi-a----- 57.00g
vm-133-disk-0 small-pool2 -wi-a----- 57.00g
vm-140-disk-0 small-pool2 -wi-a----- 57.00g
vm-163-disk-0 small-pool2 -wi-ao---- 57.00g
vm-171-disk-0 small-pool2 -wi-a----- 57.00g
vm-181-disk-0 small-pool2 -wi-ao---- 57.00g
vm-184-disk-0 small-pool2 -wi------- 57.00g
vm-107-disk-0 small-pool3 -wi-a----- 57.00g
vm-115-disk-0 small-pool3 -wi-a----- 57.00g
vm-118-disk-0 small-pool3 -wi-a----- 57.00g
vm-125-disk-0 small-pool3 -wi------- 57.00g
vm-130-disk-0 small-pool3 -wi-a----- 79.00g
vm-135-disk-0 small-pool3 -wi-a----- 57.00g
vm-141-disk-0 small-pool3 -wi-a----- 57.00g
vm-161-disk-0 small-pool3 -wi-a----- 57.00g
vm-168-disk-0 small-pool3 -wi-a----- 57.00g
vm-170-disk-0 small-pool3 -wi-a----- 57.00g
vm-178-disk-0 small-pool3 -wi-a----- 57.00g
vm-102-disk-0 small-pool4 -wi-a----- 57.00g
vm-112-disk-0 small-pool4 -wi-a----- 77.00g
vm-119-disk-0 small-pool4 -wi-a----- 77.00g
vm-128-disk-0 small-pool4 -wi-a----- 57.00g
vm-136-disk-0 small-pool4 -wi-a----- 57.00g
vm-145-disk-0 small-pool4 -wi-a----- 80.00g
vm-156-disk-0 small-pool4 -wi-a----- 57.00g
vm-157-disk-0 small-pool4 -wi------- 57.00g
vm-166-disk-0 small-pool4 -wi-a----- 57.00g
vm-175-disk-0 small-pool4 -wi-a----- 57.00g
vm-183-disk-0 small-pool4 -wi------- 57.00g
vm-109-disk-0 small-pool5 -wi-a----- 57.00g
vm-121-disk-0 small-pool5 -wi-a----- 57.00g
vm-122-disk-0 small-pool5 -wi-a----- 57.00g
vm-144-disk-0 small-pool5 -wi------- 57.00g
vm-150-disk-0 small-pool5 -wi-a----- 57.00g
vm-154-disk-0 small-pool5 -wi-a----- 57.00g
vm-155-disk-0 small-pool5 -wi-ao---- 57.00g
vm-158-disk-0 small-pool5 -wi-a----- 57.00g
vm-174-disk-0 small-pool5 -wi-a----- 57.00g
vm-180-disk-0 small-pool5 -wi-a----- 57.00g
vm-182-disk-0 small-pool5 -wi------- 57.00g
vm-186-disk-0 small-pool5 -wi------- 57.00g
vm-100-disk-0 template-pool -wi-a----- 200.00g
vm-101-disk-0 template-pool -wi-a----- 90.00g
vm-103-disk-0 template-pool -wi-a----- 90.00g
vm-104-disk-0 template-pool -wi-a----- 57.00g
vm-106-disk-0 template-pool -wi-a----- 57.00g
vm-117-disk-0 template-pool -wi-a----- 57.00g


I would appreciate your help!
Thanks in advance.
 

Attachments

  • lvm.conf.txt
    101.4 KB · Views: 4

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!