Scheduled backup fails consistently

[greavette]

Hello,

I running Proxmox 4.1-15. I've found that if I schedule a backup job either in a cluster or on a standarlone proxmox host (all using the same version), my scheduled backup fails with errors. The VM should then restart but that fails as well. When I wake up the next morning to check the status of the backup and find the VM down, I can successfuly restart the VM manually through the Proxmox GUI.

Here is the log of the backup job. I can't find what the problem is? I've been having this problem for a very long time. but it's not consistent across all my VMs. Seems to be hit and miss. Sometimes my VM backups and starts successfully after the backup...but most times it fails.

Any ideas what I can do to fix this?


INFO: starting new backup job: vzdump 107 --compress lzo --quiet 1 --mode stop --node prox4 --storage NAS --mailnotification always --mailto myemail@gmail.com
INFO: Starting Backup of VM 107 (qemu)
INFO: status = running
INFO: update VM 107: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: stopping vm
INFO: creating archive '/mnt/pve/NAS/dump/vzdump-qemu-107-2016_04_09-07_45_02.vma.lzo'
INFO: starting kvm to execute backup task
Failed to create message: Input/output error
INFO: restarting vm
INFO: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 107 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 107 -chardev 'socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/107.pid -daemonize -name VMLIMS -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga std -vnc unix:/var/run/qemu-server/107.vnc,x509,password -no-hpet -cpu 'kvm64,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce' -m 12288 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f76e6ae4957f' -drive 'file=/var/lib/vz/images/107/vm-107-disk-2.raw,if=none,id=drive-ide1,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/107/vm-107-disk-1.raw,if=none,id=drive-ide0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=101' -netdev 'type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=0E:57:B35:CF:7B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
INFO: Failed to create message: Input/output error
command 'qm start 107 --skiplock' failed: exit code 255
ERROR: Backup of VM 107 failed - start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 107 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 107 -chardev 'socket,id=qmp,path=/var/run/qemu-server/107.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/107.pid -daemonize -name VMLIMS -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga std -vnc unix:/var/run/qemu-server/107.vnc,x509,password -no-hpet -cpu 'kvm64,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce' -m 12288 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:f76e6ae4957f' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/107/vm-107-disk-1.raw,if=none,id=drive-ide0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/var/lib/vz/images/107/vm-107-disk-2.raw,if=none,id=drive-ide1,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1' -netdev 'type=tap,id=net0,ifname=tap107i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=0E:57:B35:CF:7B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard' -S' failed: exit code 1
INFO: Backup job finished with errors
TASK ERROR: job errors


Thank you.

 

[tom]

Please upgrade to latest version and test again.

If you still see the issue, please post the config of your VM

> qm config 107

 

[greavette]

Thank you Tom for the reply. I've been having this trouble with this particular VM...and sporadically with other VM's since starting to use Proxmox back in version 2. I've always hoped that an update would fix the problem but here I am at version 4.1 and still the same problem. I will plan an upgrade of my Proxmox Version in the coming weeks but hopefully with this forums help I can finally put to rest why this failing backup keeps occuring on Proxmox.

Here is the results of running qm config 107:

bootdisk: ide0
cores: 2
description: Lims Primary
ide0: local:107/vm-107-disk-1.raw,cache=writethrough,size=80G
ide1: local:107/vm-107-disk-2.raw,cache=writethrough,size=100G
ide2: none,media=cdrom
memory: 12288
name: VMLIMS
net0: e1000=0E:57:B35:CF:7B,bridge=vmbr0
numa: 0
ostype: win7
sockets: 1
startup: order=1,up=120,down=120

Thank you.

 

[dison4linux]

For what its worth I'm having a similar issue with LXC backups. The actual backup portion of the job seems to always complete, but when its supposed to start the container back up it is only successful about 75% of the time. I can then immediately power the container on myself and it starts right up.

The relevant output from the task is:

INFO: delete old backup '/mnt/pve/ns525190-backup/dump/vzdump-lxc-1009-2016_03_05-22_36_46.tar.gz'
INFO: restarting vm
INFO: lxc-start: lxc_start.c: main: 344 The container failed to start.
INFO: lxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.
INFO: lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.
command 'lxc-start -n 1009' failed: exit code 1
INFO: Finished Backup of VM 1009 (00:00:56)