SERIOUS VZDump errors

[gkovacs]

We have several 50GB+ OpenVZ VE's on one of our nodes, and for a couple of weeks now VZDump is simply unable to make backups of them most of the time.
We are making snapshot backups to NFS.

The log file is full of messages like this:

Dec 07 01:54:57 INFO: tar: ./var/clients/.../FOIB.jpg: [COLOR=#ff0000]Warning: Read error at byte 0, while reading 237 bytes: Input/output error[/COLOR]

or this:

Dec 07 01:55:11 INFO: tar: ./var/clients/.../confirm.xml: [COLOR=#ff0000]Warning: Cannot stat: Input/output error[/COLOR]

and also lots of this:

Dec 07 01:55:20 INFO: tar: ./var/clients/.../link.htm: [COLOR=#ff0000]Warning: Cannot stat: No such file or directory[/COLOR]

The files are of course there, there is no problem opening them under regular use.
It looks like the problem is somehow connected to the size of the VE, because smaller (under 20GB or so) VE backups never produce these errors.

 

[tom]

snapshot is running out of space? post the backup log.

 

[gkovacs]

I don't really understand how could a 2 GB snapshot run out of space, no one is writing ANYTHING to this VE in the middle of the night.

The error log is 250MB, here is the end of it:

Dec 07 01:56:50 INFO: tar: ./lib/libm-2.7.so: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./lib/libncurses.so.5: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./lib/libdl-2.7.so: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./lib/libatm.so.1.0.0: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./lib/libconsole.so.0.0.0: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./lib/libcidn.so.1: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./lib/libgpg-error.so.0: Warning: Cannot stat: No such file or directory
Dec 07 01:56:50 INFO: tar: ./media/: Warning: Cannot savedir: Input/output error
Dec 07 01:56:50 INFO: tar: ./media: Warning: Cannot close: Bad file descriptor
Dec 07 01:56:50 INFO: Total bytes written: 16714219520 (16GiB, 3.7MiB/s)
Dec 07 01:57:19 INFO: archive file size: 15.57GB
Dec 07 01:57:19 INFO: delete old backup '/mnt/pve/backup-daily/vzdump-openvz-102-2011_12_03-23_57_04.tar'
Dec 07 01:57:27 INFO:[COLOR=#ff0000]   /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error[/COLOR]
Dec 07 01:57:27 INFO:   Logical volume "vzsnap-proxmox-0" successfully removed
Dec 07 01:57:27 INFO: Finished Backup of VM 102 (02:04:24)

The size of the VE is about 60 GBytes, so a 16 GB backup is not going to cut it.

 

[tom]

How big is your snapshot (maximum size)? monitor the snapshot and see if it running full.

 

[gkovacs]

How big is your snapshot (maximum size)? monitor the snapshot and see if it running full.

Last night I set snapshot size to 1GB, because pvs / vgs showed 1.99GB PFree / VFree space.

Today, for no apparent reason, they show 3.99 GB PFree / VFree, so I set it to 3GB.

# cat /etc/vzdump.conf
size: 3072
bwlimit: 65536

Will monitor snapshot free space with lvs during the backup tonight.

 

[alain]

I already saw the same behavior. I think the reason is that the backup is using the free PE of the volume group for the snapshot. When the snaphot is removed after the backup, you come back to 4 GB PE free.

The message you oultined in red :
[code)
/dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
[/code]

for me indicate that no more free PE are available, so you don't have enough free space for the snapshot to backup your VM. Not only it takes time to backup 60 GB, and meanwhile lot of changes arise inside your VM, but moreover, the bigger your VM is, the more likely there will be changes inside.

Conclusion : you need more than 4 GB free PE to backup your VM. You can try also to speed the time it takes for the backup by increasing the speed of your network. Is it Gigabit to your backup storage ?

Alain