error with cfs lock '****': unable to create image: got lock timeout - aborting command

[Kenneth Sean Wirebaugh]

Hello,

I'm attempting to add a secondary virtual HDD to attach to one of my VMs, created from a storage space mounted and attached as a directory.

If I choose for it to be a small storage space like 40GB it creates it successfully, but if I try for it to be 1TiB it fails and I get this error:

update VM 300: -scsi1 storage-2:1000,format=qcow2
TASK ERROR: error with cfs lock 'storage-storage-2': unable to create image: got lock timeout - aborting command

I did attempt to find the cli command for doing this but I was not successful at locating the command, I though that may have better results.

~Sean

 

[Kenneth Sean Wirebaugh]

So Update I decided to try creating the secondary drive space using .raw instead of .qcow it worked just fine doing that.. not sure about the why but.. that will do- acceptable for me..

 

[Enrique Garcia Polo]

Same problem when I try to create a new virtual machine with a hard drive >= 50Gb
I used to create the machine with a 32Gb disk. When the machine is created, I resize the disk to 50Gb.

 

[oguz]

hi,

can't reproduce this here, can you post 'pveversion -v'?

what is the underlying storage type?

do you see anything in syslog/journal when the error appears?

 

[Enrique Garcia Polo]

The Storage is NFS

The syslog trace is:

Apr 30 09:38:04 prox01 pvedaemon[10020]: VM 998 creating disks failed
Apr 30 09:38:04 prox01 pvedaemon[10020]: unable to create VM 998 - error with cfs lock 'storage-NFScb1': unable to create image: got lock timeout - aborting command
Apr 30 09:38:04 prox01 pvedaemon[12903]: <root@pam> end task UPID:dvall-prox01:00002724:168A03F1:5EAA8020:qmcreate:998:root@pam: unable to create VM 998 - error with cfs lock 'storage-NFScb1': unable to create image: got lock timeout - aborting command
Apr 30 09:38:55 dvall-prox01 pvedaemon[22941]: <root@pam> successful auth for user 'root@pam'

The result of pvversion -v:

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.0: 6.0-11
pve-kernel-4.15: 5.4-8
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-4.15.18-20-pve: 4.15.18-46
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-22
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[fabian]

we have a hard timeout of 60s for any operation obtaining a cluster lock, which includes volume allocation on shared storages.

 

[Enrique Garcia Polo]

Is there any option to change that timeout period?

 

[fabian]

no, it's hard-coded to ensure a blocking call releases the lock again. you can manually create the image (with qemu-img create and then rescan to reference it as unused volume in the configuration

 

[Enrique Garcia Polo]

Same problem when I try to import a ovf:

# qm importovf 222 Serv-XXXX-Sistemas.ovf NFScb1 --format qcow2
import failed - error during cfs-locked 'storage-NFScb1' operation: unable to create image: got lock timeout - aborting command

In raws format there is no problem

How can I do this import?

I want qcow2 format because it can be snapshoted

 

[fabian]

your storage is simply too slow when allocating bigger images it seems. you need to manually allocate them, for example using qemu-img create or convert.

 

[Juan Pablo Abuyeres]

Hi, I have to resurrect this topic. One of our VMs has a 5Tb raw virtual drive. When trying to move that disk to another storage, the hipervisor issues this command:

/usr/bin/qemu-img create -o preallocation=metadata -f qcow2 /mnt/pve/SN7/images/373/vm-373-disk-0.qcow2 5368709120K

That takes more than 60 seconds through NFS. Preallocation takes its time I guess. It is connected via 10Gbps ethernet, which is more than enough in our case for production. So, we are kind of forced to use raw format if we need to move this drive. Any solution for this?

Thank you.

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-4.15: 5.4-12
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-2
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

 

[fabian]

upgrade (first to latest 6.4, then to 7.x) - then you can configure the preallocation mode to avoid this issue.

 

[Juan Pablo Abuyeres]

Great! Thank you!

 

[Edy Willian]

Hi,
In version 7.1 I am having the following error when creating a 2Tb virtual disk:

Formatting '/vms/images/102/vm-102-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=2147483648000 lazy_refcounts=off refcount_bits=16
TASK ERROR: unable to create VM 102 - unable to create image: 'storage-VMS'-locked command timed out - aborting

What is the solution for this error? Using the web administration panel.

 

[Juan Pablo Abuyeres]

https://pve.proxmox.com/wiki/Storage

Look for "Preallocation mode"

 

[ZmBondar]

Hi !
Virtual Environment 7.2-4

unable to create VM 115 - API Return-Code: 500. Message: Could not set allow-two-primaries on resource definition vm-115-disk-2, because: 'storage-Sql1C'-locked command timed out - aborting at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 364.

drbd: Sql1C
<------>resourcegroup Sql1C
<------>content images,rootdir
<------>controller lc
#<----->preallocation full

Jun 30 09:52:16 sh5 pvestatd[2321]: file /etc/pve/storage.cfg line 24 (section 'Sql1C') - unable to parse value of 'preallocation': unexpected property 'preallocation'

Timeout of 60s few.
The timeout occurs regularly. Especially complicates the work when restoring from archives.

 

[fabian]

preallocation is only valid for directory type storages - you seem to be using Linbit's linstor plugin? I think the next step would be to find out which operation exactly takes too long - and then why

 

[piku]

Old thread but I have a similar issue. In my case the drbd/linstor plugin times out due to the zfs backing pool operations being very slow. This is because my zfs on spinning rust pools have lots of snapshots. It's not really linstor's fault. It'd be nice to be able to adjust the timeout for storage operations somehow.

 

[piku]

Actually I found a way to work around this that is satisfactory to me. Edit /usr/share/perl5/PVE/Cluster.pm on all the cluster nodes, change alarm(60) to alarm(240) or whatever timeout you want. Then restart pvedaemon.

 

[fabian]

Actually I found a way to work around this that is satisfactory to me. Edit /usr/share/perl5/PVE/Cluster.pm on all the cluster nodes, change alarm(60) to alarm(240) or whatever timeout you want. Then restart pvedaemon.

that's really dangerous and possibly causes corruption of files in /etc/pve (which are protected by that lock and timeout).