Gluster serious issue restoring VM's

[andrea68]

I have a 4 nodes with gluster and similar problems... reported also in this thread:

In my case, I cannot restore any VM backupped to the gluster volume.
The complete error is:

Proxmox
Virtual Environment 6.4-4
Search
Storage 'BACKUP' on node 'pve01'
Search:
103
Server View
Logs
()
restore vma archive: zstd -q -d -c /mnt/pve/BACKUP/dump/vzdump-qemu-103-2021_05_08-20_39_39.vma.zst | vma extract -v -r /var/tmp/vzdumptmp28167.fifo - /var/tmp/vzdumptmp28167
CFG: size: 478 name: qemu-server.conf
DEV: dev_id=1 size: 1998236680192 devname: drive-virtio0
CTIME: Sat May  8 20:39:41 2021
Formatting 'gluster://stor01/DATA/images/107/vm-107-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=1998236680192 lazy_refcounts=off refcount_bits=16
[2021-05-08 19:25:31.426287] E [MSGID: 108006] [afr-common.c:5314:__afr_handle_child_down_event] 0-DATA-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
no lock found trying to remove 'create'  lock
error before or during data restore, some or all disks were not completely restored. VM 107 state is NOT cleaned up.
TASK ERROR: command 'set -o pipefail && zstd -q -d -c /mnt/pve/BACKUP/dump/vzdump-qemu-103-2021_05_08-20_39_39.vma.zst | vma extract -v -r /var/tmp/vzdumptmp28167.fifo - /var/tmp/vzdumptmp28167' failed: unable to create image: 'storage-DATA'-locked command timed out - aborting

I'm on the latest PVE version.

The fact that I cannot restore ANY of my VM's from backup seems to me a very serious issue.
There is nothing I can do about it?

Thanks.

 

[dcsapak]

it seems that the image creation part times out... can you try restoring to a different storage ?
if that also fails, can you copy the backup files to a different storage and try to restore from there?

 

[andrea68]

I tried with a smaller VM and restore goes fine.
The problem affect a much more larger VM (250GB of hard drive).
There's a way to avoid this problem?

Tnx.

 

[dcsapak]

any logs on the gluster server? creating an empty file should not take that long...