zst problem with unpacking archive

[plastilin]

I made a backup of the virtual machine before replacing the hard drives. Copied it to USB. There were no problems. The size of the copied file was the same as the size of the created one. The virtual machine was turned off at the time of backup. When trying to restore a machine from a backup, I received the following error:

progress 71% (read 209648091136 bytes, duration 1312 sec)
_25-03_54_28.vma.zst : Decoding error (36) : Corrupted block detected
vma: restore failed - short vma extent (1745392 < 3633664)
/bin/bash: line 1: 22400 Exit 1                  zstd -q -d -c /var/lib/vz/dump/vzdump-qemu-102-2020_07_25-03_54_28.vma.zst
     22401 Trace/breakpoint trap   | vma extract -v -r /var/tmp/vzdumptmp22398.fifo - /var/tmp/vzdumptmp22398
  Logical volume "vm-102-disk-0" successfully removed
temporary volume 'local-data:vm-102-disk-0' sucessfuly removed
  Logical volume "vm-102-disk-1" successfully removed
temporary volume 'local-data:vm-102-disk-1' sucessfuly removed
no lock found trying to remove 'create'  lock
TASK ERROR: command 'set -o pipefail && zstd -q -d -c /var/lib/vz/dump/vzdump-qemu-102-2020_07_25-03_54_28.vma.zst | vma extract -v -r /var/tmp/vzdumptmp22398.fifo - /var/tmp/vzdumptmp22398' failed: exit code 133

When trying to unzip a backup manually using the command zsdt -d vzdump-qemu-102-2020_07_25-03_54_28.vma.zst
Got the following error:
_25-03_54_28.vma.zst: 99625 MB ... _25-03_54_28.vma.zst: Decoding error (36): Corrupted block detected

This deletes the created VMA file.
How can I ignore the error and not delete the generated .vma file

 

[Alwin]

This deletes the created VMA file.
How can I ignore the error and not delete the generated .vma file

man zstd

-k, --keep
keep source file(s) after successful compression or decompression. This is the default behavior.

Or pipe it to stdout. But since the file is corrupt it is very likely not complete.

 

[plastilin]

When creating a backup, Proxmox said that everything is OK, but how can it be damaged?

2020-07-25 03:54:28 INFO: Starting Backup of VM 102 (qemu)
2020-07-25 03:54:28 INFO: status = stopped
2020-07-25 03:54:28 INFO: backup mode: stop
2020-07-25 03:54:28 INFO: ionice priority: 7
2020-07-25 03:54:28 INFO: VM Name: foxtrot
2020-07-25 03:54:28 INFO: include disk 'scsi0' 'local-data:vm-102-disk-0' 25G
2020-07-25 03:54:28 INFO: include disk 'scsi1' 'local-data:vm-102-disk-1' 250G
2020-07-25 03:54:28 INFO: creating vzdump archive '/mnt/pve/local-backup/dump/vzdump-qemu-102-2020_07_25-03_54_28.vma.zst'
2020-07-25 03:54:28 INFO: starting kvm to execute backup task
2020-07-25 03:54:29 INFO: started backup task '1b057519-9c73-44cb-b2b9-82ba6f7dde53'
2020-07-25 03:54:32 INFO: status: 0% (112.9 MiB of 275.0 GiB), duration 3, read: 37.6 MiB/s, write: 22.7 MiB/s
2020-07-25 03:54:46 INFO: status: 1% (2.9 GiB of 275.0 GiB), duration 17, read: 201.4 MiB/s, write: 101.6 MiB/s
2020-07-25 03:55:07 INFO: status: 2% (5.9 GiB of 275.0 GiB), duration 38, read: 146.2 MiB/s, write: 101.3 MiB/s
2020-07-25 03:56:30 INFO: status: 3% (8.4 GiB of 275.0 GiB), duration 121, read: 30.8 MiB/s, write: 26.3 MiB/s
2020-07-25 03:57:58 INFO: status: 4% (11.0 GiB of 275.0 GiB), duration 209, read: 31.0 MiB/s, write: 29.7 MiB/s
2020-07-25 03:59:01 INFO: status: 5% (13.8 GiB of 275.0 GiB), duration 272, read: 45.9 MiB/s, write: 43.3 MiB/s
2020-07-25 03:59:06 INFO: status: 6% (17.2 GiB of 275.0 GiB), duration 277, read: 690.6 MiB/s, write: 76.6 MiB/s
.....
2020-07-25 04:24:50 INFO: status: 94% (258.6 GiB of 275.0 GiB), duration 1821, read: 148.1 MiB/s, write: 141.6 MiB/s
2020-07-25 04:25:17 INFO: status: 95% (261.3 GiB of 275.0 GiB), duration 1848, read: 104.5 MiB/s, write: 97.4 MiB/s
2020-07-25 04:25:49 INFO: status: 96% (264.1 GiB of 275.0 GiB), duration 1880, read: 88.4 MiB/s, write: 86.8 MiB/s
2020-07-25 04:26:35 INFO: status: 97% (266.8 GiB of 275.0 GiB), duration 1926, read: 60.6 MiB/s, write: 57.5 MiB/s
2020-07-25 04:26:59 INFO: status: 98% (269.6 GiB of 275.0 GiB), duration 1950, read: 120.8 MiB/s, write: 116.4 MiB/s
2020-07-25 04:27:24 INFO: status: 99% (272.3 GiB of 275.0 GiB), duration 1975, read: 109.3 MiB/s, write: 106.5 MiB/s
2020-07-25 04:27:52 INFO: status: 100% (275.0 GiB of 275.0 GiB), duration 2003, read: 99.0 MiB/s, write: 94.5 MiB/s
2020-07-25 04:27:52 INFO: transferred 275.00 GiB in 2003 seconds (140.6 MiB/s)
2020-07-25 04:27:52 INFO: Backup is sparse: 38% (105.40 GiB) zero data
2020-07-25 04:27:52 INFO: stopping kvm after backup task
2020-07-25 04:27:59 INFO: archive file size: 77.17GB
2020-07-25 04:27:59 INFO: Finished Backup of VM 102 (00:33:31)

 

[Alwin]

When creating a backup, Proxmox said that everything is OK, but how can it be damaged?

What about corrupt storage? Bit flips in memory (non-ECC)? Defective backup storage? Flaky network transfer? The list could go on. It doesn't need to be the software's fault.

In the end though a good backup consists of backup, restore tests, disaster recovery plans and good documentation.

 

[plastilin]

The disks in the system are new. 1 week in work. Memory type is ECC. Backup doing on local storage not network. Why After create proxmox don`t check archive with key -t

-t, --test Test the integrity of compressed files.

This option is equivalent to --decompress --stdout except that the decompressed data is discarded instead of being written to standard output. No files are created or removed.

Or can you implement checking archives through the scheduler?

I think it would still be nice to add the md5sum result of the created archive to the log file. To check if the archive file is transferred to external media.

 

[Alwin]

The disks in the system are new. 1 week in work. Memory type is ECC. Backup doing on local storage not network. Why After create proxmox don`t check archive with key -t

You can run the integrity check separately by a cronjob on the server that holds the backup storage. And for the why, just try it out.

 

[plastilin]

You can run the integrity check separately by a cronjob on the server that holds the backup storage. And for the why, just try it out.

This is exactly what I will do in the future. In the meantime, I need to somehow pull out the data ...

 

[plastilin]

Update. I have reconstructed RAID-10 from old drives. I checked the backup created by Proxmox, located on this array, using the zstd -t command and the archive turned out to be intact. Then I copied it again to external media and checked again. The archive on the media was also intact. So the problem was not with Proxmox, but most likely at the time of transferring it to a USB HDD. Check your archives after transferring to external media, and only then carry out all the work on reinstalling Proxmox or replacing disks.

 

[Carnyx]

Got exactly the same error message trying to restore a number of backups

tar.zst : Decoding error (36) : Restored data doesn't match checksum

Is there a way to extract without the checksum comparison?

It would seem the ZSTD 'Fast and Good' option doesn't work very reliably

 

[Trinh Pham]

I lost 4/6 VM backups because of trusting default option ZSTD "Fast and Good".
Dell R620, ECC memory, backup to local HDD. Checked and see no error with the HDD.

 

[Carnyx]

I lost 4/6 VM backups because of trusting default option ZSTD "Fast and Good".
Dell R620, ECC memory, backup to local HDD. Checked and see no error with the HDD.

Personally I think this option should be removed. I've tested it on a number of different servers and different VM's and the successful restore rate renders it useless.

Don't trust it

 

[tudor-s]

https://forum.proxmox.com/threads/restore-backup-failure-zstd-decompression.74636/#post-490331