Hi all,
I mistakenly specified a local disk instead of the remote NFS mount as the target for the backup directory and promptly ran out of disk space.
Unfortunately now some of the /etc/qemu-server/XYZ.conf files are empty. Not all of them, but some of them:
I am assuming this is to do with the backup job eating up all the disk space (as the other two nodes in the cluster are fine), but still - this really shouldn't happen.
A log file from one of the (many) failed backups due to lack of disk space:
My question is - why on earth did those .conf files get blown away?
The irony of the backup job being destructive - gotta laugh (or cry)
I mistakenly specified a local disk instead of the remote NFS mount as the target for the backup directory and promptly ran out of disk space.
Unfortunately now some of the /etc/qemu-server/XYZ.conf files are empty. Not all of them, but some of them:
Code:
-rw-r--r-- 1 root root 206 Jul 18 14:40 337.conf
-rw-r--r-- 1 root root 207 Jul 18 14:43 338.conf
-rw-r--r-- 1 root root 207 Jul 18 14:46 340.conf
-rw-r--r-- 1 root root 206 Jul 18 14:48 341.conf
-rw-r--r-- 1 root root 207 Jul 18 14:51 342.conf
-rw-r--r-- 1 root root 0 Jul 18 14:53 343.conf
-rw-r--r-- 1 root root 0 Jul 18 14:55 344.conf
-rw-r--r-- 1 root root 0 Jul 18 14:57 345.conf
-rw-r--r-- 1 root root 0 Jul 18 14:59 346.conf
-rw-r--r-- 1 root root 0 Jul 18 15:01 347.conf
-rw-r--r-- 1 root root 0 Jul 18 15:03 348.conf
-rw-r--r-- 1 root root 0 Jul 18 15:05 349.conf
-rw-r--r-- 1 root root 0 Jul 18 15:07 350.conf
-rw-r--r-- 1 root root 0 Jul 18 15:09 351.conf
-rw-r--r-- 1 root root 0 Jul 18 15:11 432.conf
-rw-r--r-- 1 root root 0 Jul 18 15:13 433.conf
-rw-r--r-- 1 root root 0 Jul 18 15:25 99997.conf
-rw-r--r-- 1 root root 0 Jul 18 15:37 99998.conf
-rw-r--r-- 1 root root 0 Jul 18 15:50 99999.conf
I am assuming this is to do with the backup job eating up all the disk space (as the other two nodes in the cluster are fine), but still - this really shouldn't happen.
A log file from one of the (many) failed backups due to lack of disk space:
Code:
Jul 18 15:25:33 INFO: Starting Backup of VM 99998 (qemu)
Jul 18 15:25:33 INFO: stopped
Jul 18 15:25:33 INFO: status = stopped
Jul 18 15:25:34 INFO: backup mode: stop
Jul 18 15:25:34 INFO: bandwidth limit: 10240 KB/s
Jul 18 15:25:35 INFO: creating archive '/backup/vzdump-qemu-99998-2010_07_18-15_25_33.tgz'
Jul 18 15:25:35 INFO: adding '/backup/vzdump-qemu-99998-2010_07_18-15_25_33.tmp/qemu-server.conf' to archive ('qemu-server.conf')
Jul 18 15:25:35 INFO: adding '/dev/san-vm-os3-vg/vm-99998-disk-1' to archive ('vm-disk-ide0.raw')
Jul 18 15:37:45 INFO: 328331264 B 313.1 MB 730.2 s (12:10 min) 449673 B/s 439.13 KB/s
Jul 18 15:37:45 INFO: write: No space left on device
Jul 18 15:37:45 INFO: received signal - terminate process
Jul 18 15:37:49 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Jul 18 15:37:50 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Jul 18 15:37:50 INFO: /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Jul 18 15:37:51 ERROR: Backup of VM 99998 failed - command '/usr/lib/qemu-server/vmtar '/backup/vzdump-qemu-99998-2010_07_18-15_25_33.tmp/qemu-serve
r.conf' 'qemu-server.conf' '/dev/san-vm-os3-vg/vm-99998-disk-1' 'vm-disk-ide0.raw' |gzip |cstream -t 10485760 >/backup/vzdump-qemu-99998-2010_07_18-
15_25_33.dat' failed with exit code 2
My question is - why on earth did those .conf files get blown away?
The irony of the backup job being destructive - gotta laugh (or cry)