[SOLVED] Unable to create image: 'storage-NAS'-locked command timed out - aborting

[Badkey]

Why do I see this ?

One NAS 700Gb to an other NAS.

scsi0 NAS
QEMU image format (qcow2)
create full clone of drive scsi0 (QNAP:124/vm-124-disk-0.qcow2)
Formatting '/mnt/pve/NAS/images/124/vm-124-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=751619276800 lazy_refcounts=off refcount_bits=16
TASK ERROR: storage migration failed: unable to create image: 'storage-NAS'-locked command timed out - aborting

Regards John

 

[fiona]

Hi,
I'm guessing this is because with qcow2 Proxmox VE defaults to allocating metadata for the image which can take a while for network storages and big enough images. In a future version (recently applied in git and will be available with libpve-storage-perl >= 7.0-13) you'll be able to turn off preallocation.

 

[Badkey]

Ok how canwe fix this now ?

Regards John

 

[fiona]

The new storage package is currently available in the pvetest repository. If you don't want to install it now, you can try:

• If there are no snapshots on the image, you can convert it to raw format before cloning.
• Otherwise, try reducing the load as much as possible and hope you don't hit the timeout.

 

[Badkey]

Here no load. Just a 16Gb Host and 2 VM's. Try to create a VM with 700Gb image.

What do you mean by 'don't hit the timeout." ??? I use a NAS with 4 SSD's for a while. https://twitter.com/willemsej

What other information can us help to solve this ?

Regards John

 

[fiona]

For locked storage operations there is a hardcoded 60 second timeout, because the cluster file system automatically releases locks after a while (to prevent deadlocks). The locks are necessary to guarantee that different operation on the storage do not clash with each other (e.g. guarantee that a disk name is not already used).

I don't see an easy workaround except the two described above: either use the new feature to turn off preallocation or convert the disk to raw before cloning.

 

[Badkey]

At the moment I run,

Kernel Version Linux 5.4.140-1-pve #1 SMP PVE 5.4.140-1 (Wed, 08 Sep 2021 16:21:59 +0200)
PVE Manager Version pve-manager/6.4-13/9f411e79

How can I use the 'new feature to turn off preallocation'

Regards John

 

[fiona]

At the moment I run,

Kernel Version Linux 5.4.140-1-pve #1 SMP PVE 5.4.140-1 (Wed, 08 Sep 2021 16:21:59 +0200)
PVE Manager Version pve-manager/6.4-13/9f411e79

How can I use the 'new feature to turn off preallocation'

Regards John

If you're still on 6.4, you won't be able to use that feature. Other workarounds:

• Copy the disk manually. Then copy the VM configuration and adapt to new storage/VM ID.
• Temporarily add a non-shared directory storage pointing to the network storage's mount point and clone to that storage (that should avoid the cluster lock). Then adapt the disk in the VM config to use the proper storage ID, and remove the directory storage again. Having such a duplicate storage can lead to issues with migration, because Proxmox VE will see the same disks twice, so that storage should only be used as a stop-gap measure and removed again after cloning!

 

[Badkey]

Ok thanks !