vm clone failed with clound-init error

[given]

I have some vm templates with cloud-init service. But when I try to clone vm from the templates, I always get the below failed messages.


create linked clone of drive virtio0 (cloud-pool1:base-115-disk-2)
clone base-115-disk-2: base-115-disk-2 snapname __base__ to vm-833-disk-2
2020-07-07 16:10:13.529840 7f8285c480c0 -1 did not load config file, using default settings.
2020-07-07 16:10:13.537110 7f8285c480c0 -1 Errors while parsing config file!
2020-07-07 16:10:13.537118 7f8285c480c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:13.537119 7f8285c480c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:13.537126 7f8285c480c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-07-07 16:10:13.538852 7f8285c480c0 -1 Errors while parsing config file!
2020-07-07 16:10:13.538859 7f8285c480c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:13.538860 7f8285c480c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:13.538862 7f8285c480c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
create full clone of drive scsi2 (cloud-pool1:vm-115-cloudinit)
2020-07-07 16:10:14.710526 7f8bba68b0c0 -1 did not load config file, using default settings.
2020-07-07 16:10:14.719293 7f8bba68b0c0 -1 Errors while parsing config file!
2020-07-07 16:10:14.719300 7f8bba68b0c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:14.719301 7f8bba68b0c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:14.719307 7f8bba68b0c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-07-07 16:10:14.720947 7f8bba68b0c0 -1 Errors while parsing config file!
2020-07-07 16:10:14.720957 7f8bba68b0c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:14.720958 7f8bba68b0c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:14.720961 7f8bba68b0c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
rbd: create error: (17) File exists
2020-07-07 16:10:16.296525 7fbe24d200c0 -1 did not load config file, using default settings.
2020-07-07 16:10:16.301864 7fbe24d200c0 -1 Errors while parsing config file!
2020-07-07 16:10:16.301873 7fbe24d200c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.301874 7fbe24d200c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.301880 7fbe24d200c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.303516 7fbe24d200c0 -1 Errors while parsing config file!
2020-07-07 16:10:16.303525 7fbe24d200c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.303526 7fbe24d200c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.303527 7fbe24d200c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.617215 7fc48d66d0c0 -1 did not load config file, using default settings.
2020-07-07 16:10:16.621504 7fc48d66d0c0 -1 Errors while parsing config file!
2020-07-07 16:10:16.621513 7fc48d66d0c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.621514 7fc48d66d0c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.621520 7fc48d66d0c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.623101 7fc48d66d0c0 -1 Errors while parsing config file!
2020-07-07 16:10:16.623109 7fc48d66d0c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.623110 7fc48d66d0c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2020-07-07 16:10:16.623111 7fc48d66d0c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Removing image: 1% complete...
Removing image: 2% complete...
Removing image: 3% complete...
Removing image: 4% complete...
Removing image: 5% complete...
Removing image: 6% complete...
Removing image: 7% complete...
Removing image: 8% complete...
Removing image: 9% complete...
Removing image: 10% complete...
Removing image: 11% complete...
Removing image: 12% complete...
Removing image: 13% complete...
Removing image: 14% complete...
Removing image: 15% complete...
Removing image: 16% complete...
Removing image: 17% complete...
Removing image: 18% complete...
Removing image: 19% complete...
Removing image: 20% complete...
Removing image: 21% complete...
Removing image: 22% complete...
Removing image: 23% complete...
Removing image: 24% complete...
Removing image: 25% complete...
Removing image: 26% complete...
Removing image: 27% complete...
Removing image: 28% complete...
Removing image: 29% complete...
Removing image: 30% complete...
Removing image: 31% complete...
Removing image: 32% complete...
Removing image: 33% complete...
Removing image: 34% complete...
Removing image: 35% complete...
Removing image: 36% complete...
Removing image: 37% complete...
Removing image: 38% complete...
Removing image: 39% complete...
Removing image: 40% complete...
Removing image: 41% complete...
Removing image: 42% complete...
Removing image: 43% complete...
Removing image: 44% complete...
Removing image: 45% complete...
Removing image: 46% complete...
Removing image: 47% complete...
Removing image: 48% complete...
Removing image: 49% complete...
Removing image: 50% complete...
Removing image: 51% complete...
Removing image: 52% complete...
Removing image: 53% complete...
Removing image: 54% complete...
Removing image: 55% complete...
Removing image: 56% complete...
Removing image: 57% complete...
Removing image: 58% complete...
Removing image: 59% complete...
Removing image: 60% complete...
Removing image: 61% complete...
Removing image: 62% complete...
Removing image: 63% complete...
Removing image: 64% complete...
Removing image: 65% complete...
Removing image: 66% complete...
Removing image: 67% complete...
Removing image: 68% complete...
Removing image: 69% complete...
Removing image: 70% complete...
Removing image: 71% complete...
Removing image: 72% complete...
Removing image: 73% complete...
Removing image: 74% complete...
Removing image: 75% complete...
Removing image: 76% complete...
Removing image: 77% complete...
Removing image: 78% complete...
Removing image: 79% complete...
Removing image: 80% complete...
Removing image: 81% complete...
Removing image: 82% complete...
Removing image: 83% complete...
Removing image: 84% complete...
Removing image: 85% complete...
Removing image: 86% complete...
Removing image: 87% complete...
Removing image: 88% complete...
Removing image: 89% complete...
Removing image: 90% complete...
Removing image: 91% complete...
Removing image: 92% complete...
Removing image: 93% complete...
Removing image: 94% complete...
Removing image: 95% complete...
Removing image: 96% complete...
Removing image: 97% complete...
Removing image: 98% complete...
Removing image: 99% complete...
Removing image: 100% complete...done.
TASK ERROR: clone failed: error during cfs-locked 'storage-cloud-pool1' operation: rbd create vm-833-cloudinit' error: rbd: create error: (17) File exists

 

[mira]

Please post the output of pveversion -v

 

[jforman]

This seems to be happening to me as well. I'm using the HTTP API to perform the clone.

Rough Setup: 3 node proxmox cluster using LVM over iSCSI for shared VM storage.

jforman@moon1:~$ pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

I've seen https://forum.proxmox.com/threads/proxmox-lvm-iscsi-issue.69450/ and can reboot and fix the issue, but that seems like a hack and not a solution.

Output:
2020-07-07 20:46:47,422 proxmox_ubuntu_cloud.py:89 ERROR: Task status exited NOT OK: clone failed: error during cfs-locked 'storage-lvm-vmstore' operation: lvcreate 'vg-vmstore/vm-101-cloudinit' error: Failed to activate new LV vg-vmstore/vm-101-cloudinit.. Return Value: {'starttime': 1594169197, 'upid': 'UPID:moon1:0000609E:01A583DB:5F05176D:qmclone:113:jforman@pam!token1:', 'exitstatus': "clone failed: error during cfs-locked 'storage-lvm-vmstore' operation: lvcreate 'vg-vmstore/vm-101-cloudinit' error: Failed to activate new LV vg-vmstore/vm-101-cloudinit.", 'pstart': 27624411, 'tokenid': 'token1', 'pid': 24734, 'node': 'moon1', 'user': 'jforman@pam', 'status': 'stopped', 'id': '113', 'type': 'qmclone'}.

From my own script: https://github.com/jforman/virthelper

 

[given]

I'm also using HTTP API to perform the vm clone and removal. I found the problem is the removing VM action only removes the disk image. The cloud-init file of the removed VM still exists.


The output of pveversion -v as below.

proxmox-ve: 6.2-1 (running kernel: 5.4.44-1-pve)
pve-manager: 6.2-6 (running version: 6.2-6/ee1d7754)
pve-kernel-5.4: 6.2-3
pve-kernel-helper: 6.2-3
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-4.15: 5.4-17
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.10.17-2-pve: 4.10.17-20
pve-kernel-4.4.76-1-pve: 4.4.76-94
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.19-1-pve: 4.4.19-66
ceph-fuse: 12.2.13-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-3
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-8
pve-cluster: 6.1-8
pve-container: 3.1-8
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

 

[given]

Does anyone have any idea?