Restoring large backups fails (all ever made)

[pJonathan]

Hello, I have a big problem with ProxMox. I am not able to restore my backups. I get the following error:

restore vma archive: zcat /mnt/nvme1n1/dump/vzdump-qemu-263-2023_05_07-03_23_23.vma.gz | vma extract -v -r /var/tmp/vzdumptmp31931.fifo - /var/tmp/vzdumptmp31931
CFG: size: 758 name: qemu-server.conf
DEV: dev_id=1 size: 540672 devname: drive-efidisk0
DEV: dev_id=2 size: 107374182400 devname: drive-sata0
DEV: dev_id=3 size: 4194304 devname: drive-tpmstate0-backup
CTIME: Sun May  7 03:23:23 2023
Formatting '/mnt/nvme0n1/images/103/vm-103-disk-0.raw', fmt=raw size=540672 preallocation=off
new volume ID is 'SSD_1:103/vm-103-disk-0.raw'
Formatting '/mnt/nvme0n1/images/103/vm-103-disk-1.raw', fmt=raw size=107374182400 preallocation=off
new volume ID is 'SSD_1:103/vm-103-disk-1.raw'
Formatting '/mnt/nvme0n1/images/103/vm-103-disk-2.raw', fmt=raw size=4194304 preallocation=off
new volume ID is 'SSD_1:103/vm-103-disk-2.raw'
map 'drive-efidisk0' to '/mnt/nvme0n1/images/103/vm-103-disk-0.raw' (write zeros = 0)
map 'drive-sata0' to '/mnt/nvme0n1/images/103/vm-103-disk-1.raw' (write zeros = 0)
map 'drive-tpmstate0-backup' to '/mnt/nvme0n1/images/103/vm-103-disk-2.raw' (write zeros = 0)
progress 1% (read 1073807360 bytes, duration 6 sec)
progress 2% (read 2147614720 bytes, duration 13 sec)

gzip: /mnt/nvme1n1/dump/vzdump-qemu-263-2023_05_07-03_23_23.vma.gz: invalid compressed data--format violated
vma: restore failed - short vma extent (299520 < 3408384)
/bin/bash: line 1: 31933 Exit 1                  zcat /mnt/nvme1n1/dump/vzdump-qemu-263-2023_05_07-03_23_23.vma.gz
     31934 Trace/breakpoint trap   | vma extract -v -r /var/tmp/vzdumptmp31931.fifo - /var/tmp/vzdumptmp31931
temporary volume 'SSD_1:103/vm-103-disk-2.raw' sucessfuly removed
temporary volume 'SSD_1:103/vm-103-disk-1.raw' sucessfuly removed
temporary volume 'SSD_1:103/vm-103-disk-0.raw' sucessfuly removed
no lock found trying to remove 'create'  lock
error before or during data restore, some or all disks were not completely restored. VM 103 state is NOT cleaned up.
TASK ERROR: command 'set -o pipefail && zcat /mnt/nvme1n1/dump/vzdump-qemu-263-2023_05_07-03_23_23.vma.gz | vma extract -v -r /var/tmp/vzdumptmp31931.fifo - /var/tmp/vzdumptmp31931' failed: exit code 133

This error occurs on all backups I have ever made, even backups I have successfully restored several months ago. The error also only occurs with large backups, larger than 30GB from vm-disks larger than 100GB. Smaller backups are restored without any problems.
I did uncompressed backups of all VMs on LXCs today, but I already lost two VMs ... is there any way to restore the backups?

 

[fiona]

Hi,

gzip: /mnt/nvme1n1/dump/vzdump-qemu-263-2023_05_07-03_23_23.vma.gz: invalid compressed data--format violated

this usually indicates that the file is corrupted. You could try to decompress the .vma.gz file and then restore the resulting .vma file, but most likely will get the same error while trying to decompress.

This error occurs on all backups I have ever made, even backups I have successfully restored several months ago.

And this sounds like there might be an issue with the underlying disk. Can you check if the disk is healthy with e.g. smartctl? Are there any errors in /var/log/syslog related to the disk?

 

[pJonathan]

Ok, when I try to restore the backups on other machines I get the same error ... I think that's it with the data. In the future I will no longer compress on the proxmox host and store the backups on a ZFS server.

And this sounds like there might be an issue with the underlying disk. Can you check if the disk is healthy with e.g. smartctl? Are there any errors in /var/log/syslog related to the disk?

No errors, all backups distributed on two disks are corrupted


EDIT:
I take a look as lsblk and saw this:

root@pve:~# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0    20G  0 loop
loop1     7:1    0    10G  0 loop
loop2     7:2    0    25G  0 loop
loop3     7:3    0    25G  0 loop
loop4     7:4    0    10G  0 loop
loop5     7:5    0    50G  0 loop
loop6     7:6    0    50G  0 loop
loop7     7:7    0    50G  0 loop
loop8     7:8    0   200G  0 loop
loop9     7:9    0    10G  0 loop
loop10    7:10   0    50G  0 loop
loop11    7:11   0    25G  0 loop
loop12    7:12   0    50G  0 loop
loop13    7:13   0    25G  0 loop

I haven't seen this in Proxmox yet ... maybe that has something to do with the problem ...

 

[fiona]

Ok, when I try to restore the backups on other machines I get the same error ... I think that's it with the data. In the future I will no longer compress on the proxmox host and store the backups on a ZFS server.


No errors, all backups distributed on two disks are corrupted

Can you try running a memtest on your system? Do you get the same error when you try to decompress one of the backups on a different host?

EDIT:
I take a look as lsblk and saw this:

root@pve:~# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0    20G  0 loop
loop1     7:1    0    10G  0 loop
loop2     7:2    0    25G  0 loop
loop3     7:3    0    25G  0 loop
loop4     7:4    0    10G  0 loop
loop5     7:5    0    50G  0 loop
loop6     7:6    0    50G  0 loop
loop7     7:7    0    50G  0 loop
loop8     7:8    0   200G  0 loop
loop9     7:9    0    10G  0 loop
loop10    7:10   0    50G  0 loop
loop11    7:11   0    25G  0 loop
loop12    7:12   0    50G  0 loop
loop13    7:13   0    25G  0 loop

I haven't seen this in Proxmox yet ... maybe that has something to do with the problem ...

Do you have running containers with volumes on a file-based storage? Those likely come from there.