Backups hangs with these errors.

[ozgurerdogan]

I noted that backup was taking too long so I stoped task and saw these: There is only one linux kvm in HN.

May 10 00:00:01 vztl3 vzdump[113426]: INFO: starting new backup job:  vzdump 304 --quiet 1 --mailto ozgur.com@gmail.com --mode snapshot  --compress lzo --storage YEDEKLER
May 10 00:00:01 vztl3 vzdump[113426]: INFO: Starting Backup of VM 304 (qemu)
May 10 00:00:02 vztl3 qm[113429]: <root@pam> update VM 304: -lock backup
May 10 00:00:03 vztl3 kernel: EXT3-fs: barriers disabled
May 10 00:00:03 vztl3 kernel: kjournald starting.  Commit interval 5 seconds
May 10 00:00:03 vztl3 kernel: EXT3-fs (dm-3): using internal journal
May 10 00:00:03 vztl3 kernel: EXT3-fs (dm-3): mounted filesystem with ordered data mode
May 10 00:17:01 vztl3 /USR/SBIN/CRON[114108]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 10 00:30:25 vztl3 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
May 10 00:52:23 vztl3 rrdcached[1318]: flushing old values
May 10 00:52:23 vztl3 rrdcached[1318]: rotating journals
May 10 00:52:23 vztl3 rrdcached[1318]: started new journal /var/lib/rrdcached/journal//rrd.journal.1336600343.055225
May 10 00:52:23 vztl3 rrdcached[1318]: removing old journal /var/lib/rrdcached/journal//rrd.journal.1336593143.055254
May 10 01:17:01 vztl3 /USR/SBIN/CRON[116334]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 10 01:52:23 vztl3 rrdcached[1318]: flushing old values
May 10 01:52:23 vztl3 rrdcached[1318]: rotating journals
May 10 01:52:23 vztl3 rrdcached[1318]: started new journal /var/lib/rrdcached/journal//rrd.journal.1336603943.055252
May 10 01:52:23 vztl3 rrdcached[1318]: removing old journal /var/lib/rrdcached/journal//rrd.journal.1336596743.055251
May 10 02:17:01 vztl3 /USR/SBIN/CRON[118560]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 10 02:25:02 vztl3 pvedaemon[1450]: <root@pam> successful auth for user 'root@pam'
[COLOR=#ff0000]May 10 02:25:42 vztl3 kernel: Buffer I/O error on device dm-3, logical block 45351426
May 10 02:25:42 vztl3 kernel: lost page write due to I/O error on dm-3
May 10 02:25:42 vztl3 kernel: JBD: I/O error detected when updating journal superblock for dm-3.
May 10 02:25:42 vztl3 kernel: Buffer I/O error on device dm-3, logical block 45351426
May 10 02:25:42 vztl3 kernel: lost page write due to I/O error on dm-3
May 10 02:25:42 vztl3 kernel: JBD: I/O error detected when updating journal superblock for dm-3.
May 10 02:25:42 vztl3 kernel: Buffer I/O error on device dm-3, logical block 0
May 10 02:25:42 vztl3 kernel: lost page write due to I/O error on dm-3
May 10 02:25:42 vztl3 kernel: EXT3-fs (dm-3): I/O error while writing superblock[/COLOR]
May  10 02:25:43 vztl3 vzdump[113426]: ERROR: Backup of VM 304 failed -  command '/usr/lib/qemu-server/vmtar   '/YEDEKLER/dump/vzdump-qemu-304-2012_05_10-00_00_01.tmp/qemu-server.conf'  'qemu-server.conf' '/mnt/vzsnap0/images/304/vm-304-disk-1.raw'  'vm-disk-ide0.raw'|lzop  >/YEDEKLER/dump/vzdump-qemu-304-2012_05_10-00_00_01.tar.dat' failed:  interrupted by signal
May 10 02:25:43 vztl3 vzdump[113426]: INFO: Backup job finished with errors
May 10 02:25:43 vztl3 vzdump[113426]: job errors

 

[e100]

The first thing that comes to mind is that your snapshot likely ran out of space.
You can edit /etc/vzdump.conf to increase the snapshot size or set it in the GUI on the storage you are backing up from.

 

[ozgurerdogan]

Thanks for answer. But last night back was ok:

INFO: starting new backup job: vzdump 304 --remove 0 --mode snapshot --compress lzo --storage YEDEKLER --node vztl3
INFO: Starting Backup of VM 304 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO:   Logical volume "vzsnap-vztl3-0" created
INFO: creating archive '/YEDEKLER/dump/vzdump-qemu-304-2012_05_10-02_40_09.tar.lzo'
INFO: adding '/YEDEKLER/dump/vzdump-qemu-304-2012_05_10-02_40_09.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/mnt/vzsnap0/images/304/vm-304-disk-1.raw' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 161061276160 (48.15 MiB/s)
INFO: archive file size: 22.76GB
INFO: Finished Backup of VM 304 (00:53:15)
INFO: Backup job finished successfully
TASK OK

Do I still need to play with snapshot size? BTW what is it for and why I may need to play with it in /etc/vzdump.conf ?

Thanks in advance.

 

[e100]

The snapshot running out of space likely was your issue. I suggest using something larger than the default of 1024MB.

The snapshot space is used to store the data that gets changed on the source. When writes are done on the source the original data is copied to the snapshot space before it is overwritten, this is called Copy On Write(COW).

If the snapshot is 1GB then only 1GB of data can change in the source before the snapshot becomes full and stops working.

 

[ozgurerdogan]

The snapshot running out of space likely was your issue. I suggest using something larger than the default of 1024MB.

The snapshot space is used to store the data that gets changed on the source. When writes are done on the source the original data is copied to the snapshot space before it is overwritten, this is called Copy On Write(COW).

If the snapshot is 1GB then only 1GB of data can change in the source before the snapshot becomes full and stops working.

Thanks for explanation. So I should comment out #size: MB in vzdump.conf and set it like 2048MB or even more?