Cannot clone template with cloudinit drive from/to local storage

jenzopr · Oct 8, 2019

Dear all,

we're trying to clone a template with a cloudinit drive to local storage and get a TASK ERROR: clone failed: error with cfs lock 'storage-local': unable to parse volume filename 'vm-111-cloudinit'. When removing the cloudinit drive from the template, the clone works.

On a second server with the same setup, but target storage as iSCSI, the clone works without problems.

Any help is appreciated!
Best,
Jens

The output of pveversion -v is:
proxmox-ve: 5.4-2 (running kernel: 4.15.18-21-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-9
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-55
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

Richard · Oct 14, 2019

jenzopr said:
we're trying to clone a template with a cloudinit drive to local storage and get a TASK ERROR: clone failed: error with cfs lock 'storage-local': unable to parse volume filename 'vm-111-cloudinit'. When removing the cloudinit drive from the template, the clone works.

On a second server with the same setup, but target storage as iSCSI, the clone works without problems.

See https://bugzilla.proxmox.com/show_bug.cgi?id=2417

jenzopr · Oct 14, 2019

Nice! Thanks a lot, Richard.

xcdr · Oct 12, 2020

Hi @Richard , the same issue exists on NFS storage, can You fix it?

UPDATE: Problem also exists on local storage and doesn't exists on CEPH (PVE 6.2-12)

fabian · Oct 13, 2020

https://lists.proxmox.com/pipermail/pve-devel/2020-September/045287.html

xcdr · Oct 13, 2020

fabian said:
https://lists.proxmox.com/pipermail/pve-devel/2020-September/045287.html

Hi, I manually applied line changed in patch but without success, problem still exists when cloning or restoring backup file.

Bash:

TASK ERROR: clone failed: error during cfs-locked 'storage-NFS-STORAGE' operation: unable to parse volume filename 'vm-10004-cloudinit'

There is described the same issue: https://github.com/Telmate/terraform-provider-proxmox/issues/86

fabian · Oct 13, 2020

did you apply both patches in the series? could you include your pveversion -v output as well? (@mira)

xcdr · Oct 13, 2020

Yes, both patches applied.

But You should see this:

Bash:

# grep -irn "unable to parse volume filename" /usr/share/perl5/PVE
/usr/share/perl5/PVE/Storage/Plugin.pm:474:    die "unable to parse volume filename '$name'\n";
/usr/share/perl5/PVE/Storage/GlusterfsPlugin.pm:161:    die "unable to parse volume filename '$name'\n";

Software versions:

Bash:

# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
ceph: 14.2.11-pve1
ceph-fuse: 14.2.11-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

xcdr · Nov 5, 2020

Hi, bug still exists, I can't restore from backup or clone any VM that's contains cloud-init drive.

@mira There is any progress?

mira · Nov 6, 2020

I can't reproduce the issue here using a directory storage. Please provide the VM config (qm config <vmid>) as well as the output of pveversion -v.

xcdr · Nov 6, 2020

mira said:
I can't reproduce the issue here using a directory storage. Please provide the VM config (qm config <vmid>) as well as the output of pveversion -v.

Hi @mira .

I posted pveversion -v above, on current pve-manager: 6.2-15 behavior is the same.

Problem exists on local storage (/var/lib/vz) and on NFS storage, no matter if I try VM clone or restore from backup.

Message:

Code:

TASK ERROR: clone failed: error during cfs-locked 'storage-NFS-HDD' operation: unable to parse volume filename 'vm-9998-cloudinit'

Problem is reproductible on all our clusters, but not for RBD storage.

This is qm config:

Bash:

root@D001:~# qm config 101
agent: 1,fstrim_cloned_disks=1
boot: cdn
bootdisk: scsi0
cipassword: **********
ciuser: xxx
cores: 6
hotplug: disk,network,usb,cpu
ide2: NFS-HDD:101/vm-101-cloudinit.raw,media=cdrom
ipconfig0: ip=10.123.223.82/23,gw=10.123.223.22
ipconfig1: ip=10.100.223.82/24
memory: 20480
name: PHP-31
nameserver: 8.8.8.8
net0: virtio=36:A5:F7:AD:09:82,bridge=vmbr0,queues=4,tag=460
net1: virtio=2A:0D:4D:26:71:E0,bridge=vmbr0,queues=4,tag=461
numa: 0
onboot: 1
ostype: l26
scsi0: NFS-HDD:101/vm-101-disk-0.raw,size=10G
searchdomain: example.net
smbios1: uuid=733a84cd-53ae-41d7-99a6-92f26672a754
sockets: 2
tablet: 0

At the moment I must remove cloudinit drive before doing backup or clone of VM and attach it after clone or restore.

mira · Nov 6, 2020

Thank you for the VM config and please post your current versions (pveversion -v) so we know which package versions you're using exactly.

xcdr · Nov 6, 2020

mira said:
Thank you for the VM config and please post your current versions (pveversion -v) so we know which package versions you're using exactly.

Code:

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-4
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-18
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

mira · Nov 6, 2020

Have you rebooted the node since the upgrades or did you restart pveproxy.service and pvedaemon.service since then?

Did you start the clone tasks via GUI? If so, what happens when you do the clone via qm?

xcdr · Nov 6, 2020

mira said:
Have you rebooted the node since the upgrades or did you restart pveproxy.service and pvedaemon.service since then?

Did you start the clone tasks via GUI? If so, what happens when you do the clone via qm?

Yes, rebooted, problem also exists on new cluster build about one week ago.

qm clone/restore have the same issue, look at this:

Bash:

# grep -irn "unable to parse volume filename" /usr/share/perl5/PVE
/usr/share/perl5/PVE/Storage/Plugin.pm:474:    die "unable to parse volume filename '$name'\n";
/usr/share/perl5/PVE/Storage/GlusterfsPlugin.pm:161:    die "unable to parse volume filename '$name'\n";

It looks as parse_name_dir() is invoked with vm-9998-cloudinit.raw as parameter and die at line 474.

mira · Nov 9, 2020

I could reproduce it now. Looks like using raw format for the cloud-init disk is the problem. With .qcow2 format it worked.
I will look into it, thanks for reporting this.

xcdr · Nov 10, 2020

mira said:
I could reproduce it now. Looks like using raw format for the cloud-init disk is the problem. With .qcow2 format it worked.
I will look into it, thanks for reporting this.

Yes, I can confirm Your observations at current package versions.

xcdr · Nov 21, 2020

Hi @mira, what is current status of this bug?

mira · Dec 1, 2020

I sent a preliminary patch to the pve-devel list: https://lists.proxmox.com/pipermail/pve-devel/2020-December/046295.html

xcdr · Dec 2, 2020

mira said:
I sent a preliminary patch to the pve-devel list: https://lists.proxmox.com/pipermail/pve-devel/2020-December/046295.html

OK, tnx.

I see modified clone_disk() function, this patch is also correct for scenario when VM with .raw disk is restored from backup?

In the previous post I wrote also about problems with restore VM from backup, this problem occurs only for backups with .raw cloud-init drives, not for .qcow2.

Cannot clone template with cloudinit drive from/to local storage

New Member

Renowned Member

New Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Member

Proxmox Staff Member

Member