OpenVZ backup issue

[Deleted member 22531]

Hello guys,

I have an issue with ONLINE backup (snapshot) operation using Proxmox v2.x. Offline backup (STOP/START) works perfectly.

INFO: starting new backup job: vzdump 105 --remove 0 --mode snapshot --compress lzo --storage backup_vz --node master
INFO: Starting Backup of VM 105 (openvz)
INFO: CTID 105 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/vz-data ('/dev/vz/vzsnap-master-0')
INFO: Logical volume "vzsnap-master-0" created
INFO: creating archive '/mnt/backup_vz/dump/vzdump-openvz-105-2012_09_19-10_02_44.tar.lzo'
INFO: tar: ./home/user/Maildir/.Trash/cur/1332851729.P24486Q513.vm105\:2,Sb: Read error at byte 0, while reading 4096 bytes: Input/output error
INFO: tar: ./home/user/Maildir/.Trash/cur/1339153944.M812447P20581.vm105,S=17836\:2,: Read error at byte 0, while reading 2048 bytes: Input/output error
INFO: tar: ./home/user/Maildir/.Trash/cur/1332270794.P10158Q11645.vm105\:2,: Read error at byte 0, while reading 4096 bytes: Input/output error
INFO: tar: ./home/user/Maildir/.Trash/cur/1332270794.P10158Q11077.vm105\:2,: Read error at byte 0, while reading 4608 bytes: Input/output error
INFO: tar: ./home/user/Maildir/.Trash/cur/1337080970.M724505P13020.vm105,S=118738\:2,Sa: Cannot stat: Input/output error
INFO: tar: ./home/user/Maildir/.Trash/cur/1332270794.P10158Q9453.vm105\:2,: Read error at byte 0, while reading 9728 bytes: Input/output error
INFO: tar: ./root/.ssh/1: Cannot stat: No such file or directory
INFO: tar: ./root/.ssh/id_rsa.backup: Cannot stat: No such file or directory
INFO: tar: ./root/.ssh/id_rsa: Cannot stat: No such file or directory
INFO: tar: ./root/.ssh/id_rsa.pub: Cannot stat: No such file or directory
INFO: tar: ./root/.ssh/known_hosts: Cannot stat: No such file or directory
INFO: tar: ./root/.viminfo: Cannot stat: No such file or directory
INFO: tar: ./root/.profile: Cannot stat: No such file or directory
INFO: tar: ./root/.bash_history: Cannot stat: No such file or directory
INFO: tar: ./root/.mc: Cannot stat: No such file or directory
INFO: Total bytes written: 58142341120 (55GiB, 15MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: archive file size: 42.87GB
INFO: Finished Backup of VM 105 (01:09:30)
INFO: Backup job finished successfully
TASK OK

CT is powered by Debian Squeeze, as well as host.
The errors start at the relative end of the backup process. Some errors are "Input/Output Error", most - "No such file or directory".
Online backup (snapshot) is working fine for any other containters/VMs.
The error is not related to destination of backup (NFS or local).
Disabling ntp/updatedb/any other I/O-related daemons has no effect.

Is it possible that the default LVM snapshot size is not enough when there is many files?
Also maybe problem is related that this CT has the most files of all my VMs?

P.S. pveversion output:

pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-12-pve
proxmox-ve-2.6.32: 2.1-68
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-12-pve: 2.6.32-68
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-16
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1

 

[dietmar]

Is it possible that the default LVM snapshot size is not enough when there are that many files?

Yes, I would try to increase snapshot size.

 

[Deleted member 22531]

Yes, I would try to increase snapshot size.

Is there any way to find out / figure it without increasing the LVM snapshot size?
Currently:

Alloc PE / Size       285737 / 1.09 TiB
Free  PE / Size       363 / 1.42 GiB

And I'll repeat - any other VM performs snapshot well.

 

[dietmar]

Use 'lvs' to observe free snapshot space during backup.