qmrestore error using lzo compression

[jonemmons]

I'm running ProxMox 2.1 and I have been backing up a CentOS VM for a few weeks now and I thought everything was going ok until I had an issue and tried to restore from a recent backup. I ran the qmrestore command and immediately got this error message:
new volume ID is 'local:104/vm-104-disk-1.raw'
restore data to '/var/lib/vz/images/104/vm-104-disk-1.raw' (214748364800 bytes)
lzop: <stdin>: Compressed data violation
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
lzop: <stdin>: Compressed data violation
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
201849344 bytes copied, 1 s, 192.00 MiB/s
starting cleanup
temporary volume 'local:104/vm-104-disk-1.raw' sucessfuly removed
TASK ERROR: command 'zcat -f|tar xf /mnt/pve/VMBACKUP/dump/vzdump-qemu-102-2012_06_09-12_00_01.tar.lzo '--to-command=/usr/lib/qemu-server/qmextract --storage local'' failed: exit code 2

I had this error before when I tried to qmrestore a OpenSuse machine but was able to fix it by doing another backup and not selecting any compression. I can't do that for this vm because the disk are just way to large. I was able to fix the vm manually but I would really like to figure out why the backups are failing to restore.
pveversion -v
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-15
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1

Thanks in advance.

 

[Marvin]

I am running into the exact same issue. I believe the problem occurs when there is a disk in the archive that is larger than the root partition. I think that the tar script in the restore procedure uses the root partition as its extraction area. Purely speculative. I have not been able to solve this.

My root partition = local hardware RAID 72Gb
Image lvm partition = local hardware RAID 2.5Tb
Backup partition = iSCSI 250Gb

Some of my VM's have disks that are 250Gb in size (although only 1/3 used or less.) Those are the ones that fail with "tar: Unexpected EOF in archive". It does not matter if i compress the backup or not, it cannot be restored.

Can anyone lend a hand? Is this a bug?

 

[dietmar]

Is it possible to reproduce the bug?

 

[Marvin]

Is it possible to reproduce the bug?

Yes. Every time.

 

[dietmar]

Yes. Every time.

Would you mind to post the exact VM config (else it is hard to reproduce here)?

 

[Marvin]

Is this what you are asking for?:

root@proxmox2a:~# cat /etc/pve/qemu-server/115.conf
bootdisk: virtio0
cores: 4
ide2: local:iso/ProxmoxWindowsUtilities.iso,media=cdrom
memory: 4096
name: TSBUSQL3
net0: virtio=5A:B0:5A:02:34:88,bridge=vmbr5
net1: virtio=EE:AA:6A:40:91:67,bridge=vmbr0
onboot: 1
ostype: wxp
sockets: 1
virtio0: ImageDisk:vm-115-disk-1
virtio1: ImageDisk:vm-115-disk-2

 

[dietmar]

Yes. And what are the exact sizes of that disks?

 

[Marvin]

virtio0 = '/dev/ImageDisk/vm-115-disk-1' [66.00 GiB] inherit
virtio1 = '/dev/ImageDisk/vm-115-disk-2' [126.00 GiB] inherit

 

[jonemmons]

I was able to recreate my issue with a completely blank CentOS 6.2 VM. I also tried both gzip and lzo compression and both failed with the unexpected EOF in archive. I'm going to try with no compression now and see if that works. I hoping that if you can find a solution for Marvin it will also work for me. Here is the info of the vm I created:

root@RNPROX00:~# cat /etc/pve/qemu-server/109.conf
bootdisk: virtio0
cores: 4
ide2: local:iso/CentOS-6.2-x86_64-bin-DVD1.iso,media=cdrom
lock: backup
memory: 4028
name: CentOSTest
net0: rtl8139=9E:9C:AA:BB:B1:68,bridge=vmbr0
ostype: l26
sockets: 1
virtio0: VMDISKSDB_LVM:vm-109-disk-1 [32 GiB] inherit

 

[jonemmons]

I just wanted to give a update to my earlier post. I built a few more test machines and they all failed with the EOF archive error. Then I tried backups of Windows VM that I was under the assumption were just fine and they are failing with the same error. Now I'm really worried because I thought it was just the Linux VM's that had this problem. I have done plenty of Windows VM restores on both 1.x and 2.0 of ProxMox with no issues. Since I have upgraded to 2.1 I started seeing this problem pop up. Any help would be greatly appreciated.

 

[dietmar]

We are still unable to reproduce the problem here.

 

[tom]

pls try a new tar version (only for testing) and try to reproduce the issue.

wget http://ftp.at.debian.org/debian/pool/main/t/tar/tar_1.26-4_amd64.deb
dpkg -i tar_1.26-4_amd64.deb

 

[jonemmons]

pls try a new tar version (only for testing) and try to reproduce the issue.

wget http://ftp.at.debian.org/debian/pool/main/t/tar/tar_1.26-4_amd64.deb
dpkg -i tar_1.26-4_amd64.deb

I tried that and I still got the EOF Archive error when I tried to restore back down to the server. Is there any way to revert back to the version of tar that ProxMox 2.0 used? Thanks for the help!

 

[tom]

I am out of ideas as we cannot reproduce the issue here.

downgrade:

wget http://ftp.at.debian.org/debian/pool/main/t/tar/tar_1.23-3_amd64.deb
dpkg -i tar_1.23-3_amd64.deb

 

[Marvin]

I have NOT tried tar_1.26-4_amd64.deb. However i tried to do a fresh Backup and then Restore to an new vm (131) within the 2.1 gui:

extracting archive '/mnt/iSCSI_Backup/dump/vzdump-qemu-115-2012_06_20-16_17_16.tar.lzo'
extracting 'qemu-server.conf' from archive
extracting 'vm-disk-virtio1.raw' from archive
Rounding up size to full physical extent 126.00 GiB
Logical volume "vm-131-disk-1" created
new volume ID is 'ImageDisk:vm-131-disk-1'
restore data to '/dev/ImageDisk/vm-131-disk-1' (135291469824 bytes)
tar: write error

 

[Marvin]

Tried again to restore the same vm from a fresh backup to a new vm. I noticed the entire system starting to stutter (literally see pictures). The first picture was a 30-40 minutes into the restore. Then about 10 minutes later the stuttering got worse, connecting to vm consoles was intermittent, notice on 2nd picture all vm appeared down (but they were not) and vm names disappeared. Reboot of Proxmox is the only thing that would get it working again.

 

[e100]

We have 14 nodes in one of our Proxmox 2.1 clusters.
Each node has a hot swap SATA disk bay that we use for backups.
Each node has 4 backup disks, we rotate them weekly.
At our office we have one server that is built nearly identical to the production nodes
Every couple of months we restore the most recent backup of each VM from each backup disk as part of our backup validation process.

We have about 60 VMs x 4 sets of backup disks for a total of roughly 240 restores.
Every single restore worked flawless last week.
If it matters we are using lzo.

Our server in the office has a flaky stick of ECC RAM.
ECC kicks in and corrects the error so at this time it does not cause a problem.
The only time the error shows up is when we are doing these restores.
I suspect that if we did not have ECC ram and had a flaky stick it would cause strange issues with the restore process......

When you have these issues are there clues in syslog? dmesg? disk space problem?

 

[Marvin]

moved to: Upgrading Root disk on Dual Disk System