[SOLVED] Backup restore error

[gbrulet]

Hello,
I tried to restore a backup multiple time and all-time fail.
The backup comes from another Proxmox server.

Here is the log I have:

restore vma archive: lzop -d -c /var/lib/vz/dump/vzdump-qemu-105-2019_08_11-01_21_27.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp11512.fifo - /var/tmp/vzdumptmp11512
CFG: size: 328 name: qemu-server.conf
DEV: dev_id=1 size: 536870912000 devname: drive-scsi0
CTIME: Sun Aug 11 01:21:28 2019
disk image '/var/lib/vz/images/101/vm-101-disk-0.qcow2' does not exists
Formatting '/var/lib/vz/images/101/vm-101-disk-0.qcow2', fmt=qcow2 size=536870912000 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
new volume ID is 'local:101/vm-101-disk-0.qcow2'
map 'drive-scsi0' to '/var/lib/vz/images/101/vm-101-disk-0.qcow2' (write zeros = 0)
progress 1% (read 5368709120 bytes, duration 34 sec)
progress 2% (read 10737418240 bytes, duration 57 sec)
progress 3% (read 16106127360 bytes, duration 78 sec)
progress 4% (read 21474836480 bytes, duration 101 sec)
progress 5% (read 26843545600 bytes, duration 126 sec)
progress 6% (read 32212254720 bytes, duration 142 sec)
progress 7% (read 37580963840 bytes, duration 158 sec)
progress 8% (read 42949672960 bytes, duration 175 sec)
progress 9% (read 48318382080 bytes, duration 197 sec)
progress 10% (read 53687091200 bytes, duration 221 sec)
progress 11% (read 59055800320 bytes, duration 238 sec)
progress 12% (read 64424509440 bytes, duration 261 sec)
progress 13% (read 69793218560 bytes, duration 262 sec)
progress 14% (read 75161927680 bytes, duration 263 sec)
progress 15% (read 80530636800 bytes, duration 263 sec)
progress 16% (read 85899345920 bytes, duration 263 sec)
progress 17% (read 91268055040 bytes, duration 263 sec)
progress 18% (read 96636764160 bytes, duration 263 sec)
progress 19% (read 102005473280 bytes, duration 263 sec)
progress 20% (read 107374182400 bytes, duration 264 sec)
progress 21% (read 112742891520 bytes, duration 264 sec)
progress 22% (read 118111600640 bytes, duration 264 sec)
progress 23% (read 123480309760 bytes, duration 264 sec)
progress 24% (read 128849018880 bytes, duration 264 sec)
progress 25% (read 134217728000 bytes, duration 264 sec)
progress 26% (read 139586437120 bytes, duration 264 sec)
progress 27% (read 144955146240 bytes, duration 264 sec)
progress 28% (read 150323855360 bytes, duration 265 sec)
progress 29% (read 155692564480 bytes, duration 266 sec)
progress 30% (read 161061273600 bytes, duration 297 sec)
progress 31% (read 166429982720 bytes, duration 329 sec)
progress 32% (read 171798691840 bytes, duration 361 sec)
progress 33% (read 177167400960 bytes, duration 393 sec)
progress 34% (read 182536110080 bytes, duration 423 sec)
lzop: /var/lib/vz/dump/vzdump-qemu-105-2019_08_11-01_21_27.vma.lzo: Compressed data violation

** (process:11515): ERROR **: restore failed - short vma extent (277504 < 3797504)
/bin/bash: line 1: 11514 Exit 1 lzop -d -c /var/lib/vz/dump/vzdump-qemu-105-2019_08_11-01_21_27.vma.lzo
11515 Trace/breakpoint trap | vma extract -v -r /var/tmp/vzdumptmp11512.fifo - /var/tmp/vzdumptmp11512
temporary volume 'local:101/vm-101-disk-0.qcow2' sucessfuly removed
TASK ERROR: command 'set -o pipefail && lzop -d -c /var/lib/vz/dump/vzdump-qemu-105-2019_08_11-01_21_27.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp11512.fifo - /var/tmp/vzdumptmp11512' failed: exit code 133

 

[Chris]

Hi,
it seems that your compressed archive is corrupt.

Compressed data violation

Can you decompress it manually by running `lzop -d path`? What version of PVE are you running?

 

[gbrulet]

Hello,

Thanks for the reply.

I started the manualy decompression, I'll keep you inform asap.

I'm running Proxmox VE 5 (ZFS) on OVH server. (Virtual Environment 5.4-13)

 

[gbrulet]

The manual decompression finish with the same error

root@ns3135078:/var/lib/vz/dump# lzop -d vzdump-qemu-105-2019_08_11-01_21_27.vma.lzo
lzop: vzdump-qemu-105-2019_08_11-01_21_27.vma.lzo: Compressed data violation

 

[Chris]

A quick search in the forum reveals that such errors could be hardware (RAM) related. Please check your RAM by running a hardware test https://docs.ovh.com/gb/en/dedicated/hardware-diagnostics/

 

[Chris]

For reference:
https://forum.proxmox.com/threads/problems-on-restore-from-backup.20732/

 

[gbrulet]

Thanks for the answer !

You're right, I strarted a RAM test on my OVH server and the test crashed at 68% (bit-spread).
I'll contact OVH service and keep you informed.

 

[gbrulet]

Do I need to check the RAM on the Proxmox server that create the backup or only the one that try to restore the backup ?

 

[Chris]

Well, I would recommend on both, the error might get introduced during compression or during decompression.

 

[gbrulet]

It mean that if I create a backup without compression it should work fine during the restore ?

 

[Chris]

Well yes or no, defect RAM can introduce a lot of incorrect behavior and some very very weird errors. It is not necessarily limited to incorrect compression via lzop.

 

[gbrulet]

Ok another question, when the server make the backup it save it directly to a remote NFS storage.
Do that can "corrupt" the backup ?

 

[PlOrAdmin]

It's possible yes.

 

[gbrulet]

Hello,

Sorry for the delay.

I tested the ram on both server and everything is OK.
I'll try to do a backup one more time and test.
Else, any others idea ?

 

[PlOrAdmin]

Let us know the result of your backup test. The destination backup volumes are healthy I assume?

 

[gbrulet]

Hello,

Sorry for the delay, I just create another backup and this one worked good.

Best regards

 

[Chris]

Happy to hear that the issue is resolved. Please mark the thread as solved as tag for others.

 

[sigmarb]

Just for the next lost souls. It was no hardware issue at all for us. Solution for us is here documented:

https://www.cubewerk.de/2022/10/25/vma-restore-failed-short-vma-extent-compressed-data-violation/