Ran out of disk space during backup - now some /etc/qemu-server/XYZ.conf are empty!

yatesco

Well-Known Member
Sep 25, 2009
211
5
58
Hi all,

I mistakenly specified a local disk instead of the remote NFS mount as the target for the backup directory and promptly ran out of disk space.

Unfortunately now some of the /etc/qemu-server/XYZ.conf files are empty. Not all of them, but some of them:

Code:
-rw-r--r--  1 root root  206 Jul 18 14:40 337.conf
-rw-r--r--  1 root root  207 Jul 18 14:43 338.conf
-rw-r--r--  1 root root  207 Jul 18 14:46 340.conf
-rw-r--r--  1 root root  206 Jul 18 14:48 341.conf
-rw-r--r--  1 root root  207 Jul 18 14:51 342.conf
-rw-r--r--  1 root root    0 Jul 18 14:53 343.conf
-rw-r--r--  1 root root    0 Jul 18 14:55 344.conf
-rw-r--r--  1 root root    0 Jul 18 14:57 345.conf
-rw-r--r--  1 root root    0 Jul 18 14:59 346.conf
-rw-r--r--  1 root root    0 Jul 18 15:01 347.conf
-rw-r--r--  1 root root    0 Jul 18 15:03 348.conf
-rw-r--r--  1 root root    0 Jul 18 15:05 349.conf
-rw-r--r--  1 root root    0 Jul 18 15:07 350.conf
-rw-r--r--  1 root root    0 Jul 18 15:09 351.conf
-rw-r--r--  1 root root    0 Jul 18 15:11 432.conf
-rw-r--r--  1 root root    0 Jul 18 15:13 433.conf
-rw-r--r--  1 root root    0 Jul 18 15:25 99997.conf
-rw-r--r--  1 root root    0 Jul 18 15:37 99998.conf
-rw-r--r--  1 root root    0 Jul 18 15:50 99999.conf

I am assuming this is to do with the backup job eating up all the disk space (as the other two nodes in the cluster are fine), but still - this really shouldn't happen.

A log file from one of the (many) failed backups due to lack of disk space:

Code:
Jul 18 15:25:33 INFO: Starting Backup of VM 99998 (qemu)
Jul 18 15:25:33 INFO: stopped
Jul 18 15:25:33 INFO: status = stopped
Jul 18 15:25:34 INFO: backup mode: stop
Jul 18 15:25:34 INFO: bandwidth limit: 10240 KB/s
Jul 18 15:25:35 INFO: creating archive '/backup/vzdump-qemu-99998-2010_07_18-15_25_33.tgz'
Jul 18 15:25:35 INFO: adding '/backup/vzdump-qemu-99998-2010_07_18-15_25_33.tmp/qemu-server.conf' to archive ('qemu-server.conf')
Jul 18 15:25:35 INFO: adding '/dev/san-vm-os3-vg/vm-99998-disk-1' to archive ('vm-disk-ide0.raw')
Jul 18 15:37:45 INFO: 328331264 B 313.1 MB 730.2 s (12:10 min) 449673 B/s 439.13 KB/s
Jul 18 15:37:45 INFO: write: No space left on device
Jul 18 15:37:45 INFO: received signal - terminate process
Jul 18 15:37:49 INFO:   /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Jul 18 15:37:50 INFO:   /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Jul 18 15:37:50 INFO:   /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Jul 18 15:37:51 ERROR: Backup of VM 99998 failed - command '/usr/lib/qemu-server/vmtar '/backup/vzdump-qemu-99998-2010_07_18-15_25_33.tmp/qemu-serve
r.conf' 'qemu-server.conf' '/dev/san-vm-os3-vg/vm-99998-disk-1' 'vm-disk-ide0.raw' |gzip |cstream -t 10485760 >/backup/vzdump-qemu-99998-2010_07_18-
15_25_33.dat' failed with exit code 2

My question is - why on earth did those .conf files get blown away?

The irony of the backup job being destructive - gotta laugh (or cry)
 
Re: Ran out of disk space during backup - now some /etc/qemu-server/XYZ.conf are empt

My question is - why on earth did those .conf files get blown away?

The irony of the backup job being destructive - gotta laugh (or cry)

Right, that should not happen - I will take a careful look at the code.
 
Re: Ran out of disk space during backup - now some /etc/qemu-server/XYZ.conf are empt

I can also confirm this issue.

Code:
proxmox1:~# pveversion -v
pve-manager: 1.5-10 (pve-manager/1.5/4822)
running kernel: 2.6.32-2-pve
proxmox-ve-2.6.32: 1.5-8
pve-kernel-2.6.32-1-pve: 2.6.32-4
pve-kernel-2.6.32-2-pve: 2.6.32-8
pve-kernel-2.6.24-10-pve: 2.6.24-21
pve-kernel-2.6.24-8-pve: 2.6.24-16
qemu-server: 1.1-16
pve-firmware: 1.0-5
libpve-storage-perl: 1.0-13
vncterm: 0.9-2
vzctl: 3.0.23-1pve11
vzdump: 1.2-6
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.4-1
ksm-control-daemon: 1.0-4
 
Re: Ran out of disk space during backup - now some /etc/qemu-server/XYZ.conf are empt

I ran into this issue last night. Exact same problem. My NFS share was inactive (my fault, I brought it down for service), I ran a vzdump, and it ran out of space. Same error shown above. I have not tracked down the cause yet but I can confirm that the following steps restored my vm (Centos 5.5 KVM with RAW Image on DRBD backed LVM):
1. Recreate the vm conf file exactly as it was before it got wiped. I did this by copying a very similar conf file and naming it correctly and then modifying it appropriately. You could also just restore the conf from backup. I highly recommend backing up all PVE config files as part of your backup routine.
2. I think the only absolutely important mod to the conf would probably be your storage disks including specifying the appropriate bootdisk.
3. Your VM Ethernet Card will most likely be messed up since the pve conf file includes the card's MAC address. I just omitted the card from the conf file and created a new one. On boot, linux complained, but I was able to vnc in from the PVE web interface and reconfigure my networking. If you have the original MAC though, adding appropriate VLAN entries in the config should result in no issues booting... I assume.

Anyway, I am sure a fix is imminent but in the meantime I can confirm that this issue does not corrupt your actual VM image and that it will boot if you recreate the conf file... at least with a raw KVM image.
 
Re: Ran out of disk space during backup - now some /etc/qemu-server/XYZ.conf are empt

Hello,
we had the same problem.
Is there any solution yet?
 
Re: Ran out of disk space during backup - now some /etc/qemu-server/XYZ.conf are empt

Hello,
we had the same problem.
Is there any solution yet?

The next version will include a fix for some cases, but is is not sure that I caught all errors - the perl 'print' command is quite unsafe, and I was not aware of that fact. In 2.0 this will be fixed.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!