Proxmox 2.2 Backup Problem

[michu]

Hi,

Updated from Proxmox 2.1 to Proxmox 2.2 and I've got some backup problems.
Backup mode is snapshot, machine is KVM with raw file and virtio drivers, destination for backup is sshfs mount point.
With 2.1 never got problems.

Proxmox version:
pve-manager: 2.2-26 (pve-manager/2.2/c1614c8c)
running kernel: 2.6.32-16-pve
proxmox-ve-2.6.32: 2.2-80
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-16-pve: 2.6.32-80
pve-kernel-2.6.32-14-pve: 2.6.32-74
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-1
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-28
qemu-server: 2.0-64
pve-firmware: 1.0-21
libpve-common-perl: 1.0-37
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-34
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.2-7
ksm-control-daemon: 1.1-1

Guest version:
Ubuntu 10.04 LTS Server x64 with newest kernel 2.6.32-45-server #99-Ubuntu SMP Tue Oct 16 16:41:38 UTC 2012 x86_64 GNU/Linux

---------------------------------------------

Here are some logs:

1) Mail

Nov 14 23:23:11 INFO: ERROR: incomplete read detected
Nov 14 23:23:19 ERROR: Backup of VM 209 failed - command '/usr/lib/qemu-server/vmtar '/var/lib/vz/pvesshfs/pve1x02/daily/1-3/dump/vzdump-qemu-209-2012_11_14-22_13_14.tmp/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/209/vm-209-disk-2.raw' 'vm-disk-virtio1.raw' '/mnt/vzsnap0/images/209/vm-209-disk-1.raw' 'vm-disk-virtio0.raw'|lzop >/var/lib/vz/pvesshfs/pve1x02/daily/1-3/dump/vzdump-qemu-209-2012_11_14-22_13_14.tar.dat' failed: exit code 255

2) /var/log/messages

Nov 14 23:23:15 pve01 kernel: lost page write due to I/O error on dm-4
Nov 14 23:23:15 pve01 kernel: lost page write due to I/O error on dm-4

dm-4 does not exists when I do fdisk -l, so I assume it's LVM snapshot.


Regards,
michu

 

[michu]

Hi,

I think I have found solution.
Below is link:

http://thegioinguonmo.com/virtualization/kvm/vzdump-lvm-snapshots-kernel-errors.html

Author says: This would show up on busy servers only, probably caused due to lvm snapshot running out of space.
The solution is to increasize size for vzdump
Author edited /usr/bin/vzdump file and changed: run_command (*LOG, "$lvcreate --size 1000m --snapshot --name vzsnap /dev/$lvmvg/$lvmlv");

Question to Proxmox team: Is it the same value that can be modified via configuration file /etc/vzdump with parameter size: M ????

Regards,
michu

 

[tom]

do never edit /usr/bin/vzdump

set the needed value in /etc/vzdump.conf

and upgrade to latest version from today, including a fix for for the haning backup jobs in the case of out of space problem.

 

[michu]

Hi Tom,

Thanks for reply. I have not planned to edit /usr/bin/vzdump, that's why I asked if /etc/vzdump.conf size parameter corresponds to the size value in /usr/bin/vzdump.
One more question: What is default value for size parameter in /etc/vzdump.conf ? I mean: What default value do you use in Proxmox system ? I'll try to double this value and check if it solves problem.

Regards,
michu

 

[tom]

we use 1024 as default (1 GB). see 'man vzdump' for all details.

but note, you can increase this value only up to the total free space:

example:
__
root@mits1:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 3.63t 12.50g
__

this means I can use up 12.5 GB here.

 

[michu]

Thanks for info Tom. I've got 15.99g so I try to set size to 2048 and see what happens.
Thanks for help.

Regards,
michu

 

[jva1601]

I was testing different settings to resolve my backup failures with a large vm. I tried changing it to use gzip from lzo to see if it succeeds this too failed.
I was able to finally make the backup work after I changed the setting in vzdump.conf to use size: 2048. The backup took an hour to complete but at least
it worked!

INFO: starting new backup job: vzdump 101 --quiet 1 --mailto user@example.com --mode snapshot --compress lzo --storage LOCAL-HD-BACKUP-EXT3
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxmox-carrot-0" created
INFO: creating archive '/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-17_45_02.tar.lzo'
INFO: adding '/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-17_45_02.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.qcow2' to archive ('vm-disk-virtio0.qcow2')
INFO: Total bytes written: 171825170944 (43.20 MiB/s)
INFO: archive file size: 142.35GB
INFO: delete old backup '/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-14_45_02.tar.gz'
INFO: Finished Backup of VM 101 (01:03:25)
INFO: Backup job finished successfully
TASK OK

Failed message:
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxmox-carrot-0" created
INFO: creating archive '/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-02_31_49.tar.lzo'
INFO: adding '/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-02_31_49.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/101/vm-101-disk-1.qcow2' to archive ('vm-disk-virtio0.qcow2')
INFO: ERROR: incomplete read detected
ERROR: Backup of VM 101 failed - command '/usr/lib/qemu-server/vmtar '/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-02_31_49.tmp/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/101/vm-101-disk-1.qcow2' 'vm-disk-virtio0.qcow2'|lzop >/LOCAL-HD-BACKUP/dump/vzdump-qemu-101-2012_11_14-02_31_49.tar.dat' failed: exit code 255

 

[tom]

INFO: ERROR: incomplete read detected

=> that points to a full snapshot.

 

[jmjosebest]

Hi,

I'm having the same problem,
any idea?

INFO: Starting Backup of VM 152 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-pveabs2-0" created
INFO: creating archive '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-12_01_07.tar.lzo'
INFO: adding '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-12_01_07.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/152/vm-152-disk-1.qcow2' to archive ('vm-disk-sata0.qcow2')
INFO: ERROR: incomplete read detected
ERROR: Backup of VM 152 failed - command '/usr/lib/qemu-server/vmtar '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-12_01_07.tmp/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/152/vm-152-disk-1.qcow2' 'vm-disk-sata0.qcow2'|lzop >/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-12_01_07.tar.dat' failed: exit code 255
INFO: Backup job finished with errors
TASK ERROR: job errors

root@pveabs2:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 pve lvm2 a-- 167.18g 16.00g
/dev/sdb1 ssd2 lvm2 a-- 447.12g 7.12g
root@pveabs2:~#
root@pveabs2:~#
root@pveabs2:~# vgdisplay
--- Volume group ---
VG Name ssd2
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 18
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 447.12 GiB
PE Size 4.00 MiB
Total PE 114464
Alloc PE / Size 112640 / 440.00 GiB
Free PE / Size 1824 / 7.12 GiB
VG UUID CFt1jy-HZJc-v5Ab-ylBk-AB4z-0O24-WQYmlX


--- Volume group ---
VG Name pve
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 52
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size 167.18 GiB
PE Size 4.00 MiB
Total PE 42798
Alloc PE / Size 38703 / 151.18 GiB
Free PE / Size 4095 / 16.00 GiB
VG UUID oMzdl4-RXoY-gJ0v-VoIX-Mj3i-QmbJ-mPY751


root@pveabs2:~#
root@pveabs2:~#
root@pveabs2:~#
root@pveabs2:~# cat /etc/vzdump.conf
# vzdump default settings


#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
size: 5120
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST
root@pveabs2:~#

 

[jmjosebest]

Also same issue, doing by hand in ssh:

root@pveabs2:~# vzdump 152 --remove 0 --mode snapshot --compress lzo --storage nfsabs1-private --node pveabs2 --size 7000
INFO: starting new backup job: vzdump 152 --remove 0 --mode snapshot --compress lzo --node pveabs2 --storage nfsabs1-private --size 7000
INFO: Starting Backup of VM 152 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-pveabs2-0" created
INFO: creating archive '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-13_09_49.tar.lzo'
INFO: adding '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-13_09_49.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/152/vm-152-disk-1.qcow2' to archive ('vm-disk-sata0.qcow2')
INFO: ERROR: incomplete read detected
ERROR: Backup of VM 152 failed - command '/usr/lib/qemu-server/vmtar '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-13_09_49.tmp/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/152/vm-152-disk-1.qcow2' 'vm-disk-sata0.qcow2'|lzop >/mnt/pve/nfsabs1-private/dump/vzdump-qemu-152-2012_11_30-13_09_49.tar.dat' failed: exit code 255
INFO: Backup job finished with errors
job errors
root@pveabs2:~#

 

[jmjosebest]

Ok, I found some extra info:



Originally Posted by cesarpk [URL="http://forum.proxmox.com/showthread.php?p=64835#post64835"]
just pretend to have a rough idea and that i not feel lost in the space for futures changes



You can observe used snapshot space during backup using 'lvs'

http://forum.proxmox.com/threads/11882-Simultaneous-VMs-backups-between-two-nodes-PVE-Cluster-and-bwlimit[/URL]


Seems I need more free space on the VG...
Too expensive if I'm using SSD disks... I will try to a mixed SSD + SATA VG.

Thanks.

 

[jva1601]

I know Tom mentioned if you see a message "INFO: ERROR: incomplete read detected"

=> that points to a full snapshot.

But I also knew I had enough free space on the local storage as well as the backup storage. Through trial and error I was able to make my backups work consistently irregardless of how large a particular vm size by scheduling them to be backup separately. If I lump them all together in one backup schedule especially vm's I created with HD's as large as 150GB or more backup tends to fail with the dreaded INFO: ERROR: incomplete read detected. For vm's with smaller HD's like 30GB or less I can back them up successfully together using the same backup schedule. I also had to increase the vzdump.conf size to 2048, otherwise the backup will still fail.

Why it works now I am not entirely sure but my hunch says whenever the "vzsnap-proxmox-carrot-0" gets created on my logical volume it starts to consume all available free space. My total HD space on my local storage is only 3.8TB and it is configured for RAID 5 I know it is not good practice but for our small potatoes operation it is good enough.

 

[jmjosebest]

I know Tom mentioned if you see a message "INFO: ERROR: incomplete read detected"

=> that points to a full snapshot.

But I also knew I had enough free space on the local storage as well as the backup storage. Through trial and error I was able to make my backups work consistently irregardless of how large a particular vm size by scheduling them to be backup separately. If I lump them all together in one backup schedule especially vm's I created with HD's as large as 150GB or more backup tends to fail with the dreaded INFO: ERROR: incomplete read detected. For vm's with smaller HD's like 30GB or less I can back them up successfully together using the same backup schedule. I also had to increase the vzdump.conf size to 2048, otherwise the backup will still fail.

Why it works now I am not entirely sure but my hunch says whenever the "vzsnap-proxmox-carrot-0" gets created on my logical volume it starts to consume all available free space. My total HD space on my local storage is only 3.8TB and it is configured for RAID 5 I know it is not good practice but for our small potatoes operation it is good enough.

You can run the command "lvs" during the backup process to see how many free space you need on the VG.
I think the case is: If during the backup process, you download a file with 10GB size inside the KVM machine, and your VG free space in the HN is 7GB, the backup return error, because you need 3GB extra free space on the VG.
Then if your KVM VPS is used for webhosting, maybe with a low VG free space is Ok for you.
But if your KVM VPS is used to download via p2p many files, you need a biggest VG free space, because during the backup process the main LV is blocked and all changements are doing on the vzsnap. If your vzsnap has not free space, your vps will be inconsistient if the backup dont stops.

But I'm not really sure, maybe Tom can confirm

Regards.