Backup failed PVE 4.1

[ivix]

Hello,

i have problem with backup on PVE 4.1, randomly backup job crash and the VM has crash too.
This VM is on NFS storage but, VM locate on Ceph have same problem with backups.

INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --storage local --node c01 --compress lzo
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: update VM 101: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-101-2016_02_15-08_21_15.vma.lzo'
INFO: started backup task '1961ec71-d795-439e-9888-20a0eddbcd65'
INFO: status: 2% (176291840/8589934592), sparse 2% (173518848), duration 3, 58/0 MB/s
INFO: status: 4% (354746368/8589934592), sparse 3% (340590592), duration 6, 59/3 MB/s
ERROR: VM 101 not running
INFO: aborting backup job
ERROR: VM 101 not running
ERROR: Backup of VM 101 failed - VM 101 not running
INFO: Backup job finished with errors
TASK ERROR: job errors

VM config

boot: c
bootdisk: virtio0
cores: 1
cpu: kvm64
cpuunits: 1000
hotplug: disk,network,usb
ide2: none,media=cdrom
memory: 1024
name: Monitoring
net0: e1000=96:43:E8:9C:D5:8C,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
sockets: 1
tablet: 0
virtio0: network-raid10:101/vm-101-disk-1.raw,cache=none,format=raw,iothread=1,size=8589934592

Package versions:

proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve) 
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c) 
pve-kernel-4.2.6-1-pve: 4.2.6-34 lvm2: 2.02.116-pve2 
corosync-pve: 2.3.5-2 
libqb0: 0.17.2-1 
pve-cluster: 4.0-31 
qemu-server: 4.0-49 
pve-firmware: 1.1-7 
libpve-common-perl: 4.0-45 
libpve-access-control: 4.0-11 
libpve-storage-perl: 4.0-38 
pve-libspice-server1: 0.12.5-2 
vncterm: 1.2-1 
pve-qemu-kvm: 2.5-3 
pve-container: 1.0-39 
pve-firewall: 2.0-15 
pve-ha-manager: 1.0-19 
ksm-control-daemon: 1.2-1 
glusterfs-client: 3.5.2-2+deb8u1 
lxc-pve: 1.1.5-6 
lxcfs: 0.13-pve3 
cgmanager: 0.39-pve1 
criu: 1.6.0-1 
zfsutils: 0.6.5-pve7~jessie

 

[wolfgang]

Hi
have you try to limit the speed of the backup.
it looks like you have overload the network.

 

[ivix]

Hi,
no, I am not limit a network is not overloaded. I use 2x1Gbit Balance-rr.
Even if the network is overloaded it should not affect VM crash

I attach another example

INFO: starting new backup job: vzdump 102 --remove 0 --node c01 --mode snapshot --storage network-raid10 --compress lzo
INFO: Starting Backup of VM 102 (qemu)
INFO: status = running
INFO: update VM 102: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/network-raid10/dump/vzdump-qemu-102-2016_02_14-21_47_20.vma.lzo'
INFO: started backup task 'fc167441-716a-457f-8ea5-80ea03f28423'
INFO: status: 0% (346357760/193273528320), sparse 0% (106586112), duration 3, 115/79 MB/s
INFO: status: 1% (1947205632/193273528320), sparse 0% (149880832), duration 20, 94/91 MB/s
INFO: status: 2% (3869900800/193273528320), sparse 0% (310939648), duration 50, 64/58 MB/s
INFO: status: 3% (5857476608/193273528320), sparse 0% (362160128), duration 80, 66/64 MB/s
INFO: status: 4% (7769161728/193273528320), sparse 0% (381931520), duration 113, 57/57 MB/s
INFO: status: 5% (9680846848/193273528320), sparse 0% (406233088), duration 143, 63/62 MB/s
INFO: status: 6% (11650859008/193273528320), sparse 0% (411635712), duration 176, 59/59 MB/s
INFO: status: 7% (13598588928/193273528320), sparse 0% (413585408), duration 211, 55/55 MB/s
INFO: status: 8% (15507521536/193273528320), sparse 0% (1752166400), duration 236, 76/22 MB/s
ERROR: VM 102 not running
INFO: aborting backup job
ERROR: VM 102 not running
ERROR: Backup of VM 102 failed - VM 102 not running
INFO: Backup job finished with errors
TASK ERROR: job errors

This VM is on Ceph storage and the destination of backup is on NFS share

 

[aychprox]

I observed backup is not possible for disk image that with iothread=1.
This is what I get when try to manually backup VM:

Spoiler: backup logs

INFO: starting new backup job: vzdump 100 --compress lzo --remove 0 --mode snapshot --storage NFS --node 103C3
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: update VM 100: -lock backup
ERROR: Backup of VM 100 failed - disk 'virtio0' (iothread=on) can't use backup feature currently. Please set backup=no for this drive at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 71.
INFO: Backup job finished with errors
TASK ERROR: job errors

How to disable iothread for a running VM?

 

[ivix]

Now I updated PVE-manager: 4.1-5 to 4.1-13 and I received an error that describes. 4.1-5 do not notify the information.
And when the backup option will be implemented with iothread = 1?

 

[aychprox]

I had tried to modified 100.conf under /etc/pve/qemu-server by remove iothread=1. Backup starting working, but halfway errors and saying that vm is not running (vm crashed and shutdown)

Spoiler: backup logs

INFO: starting new backup job: vzdump 100 --mode snapshot --remove 0 --compress lzo --storage NFS --node 103C3
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: update VM 100: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating archive '/mnt/pve/NFS/dump/vzdump-qemu-100-2016_02_15-21_48_42.vma.lzo'
ERROR: VM 100 qmp command 'guest-fsfreeze-freeze' failed - got timeout
ERROR: VM 100 qmp command 'guest-fsfreeze-thaw' failed - got timeout
INFO: started backup task 'f63f020f-d07b-4b22-acf7-0d0a18e0fdfc'
INFO: status: 0% (272826368/536870912000), sparse 0% (188342272), duration 3, 90/28 MB/s
INFO: status: 1% (5390008320/536870912000), sparse 0% (889290752), duration 381, 13/11 MB/s
ERROR: VM 100 not running
INFO: aborting backup job
ERROR: VM 100 not running
ERROR: Backup of VM 100 failed - VM 100 not running
INFO: Backup job finished with errors
TASK ERROR: job errors

== Update ==
after change the conf file, reboot server, backup working fine.

 

[Alexander Schmid]

Hello,
i currently evaluate proxmox and do so some tests with different storage-options.

I wonder what backup-options for VM's that have Disks with iothread/dataplane on a blockbased storage actually exists ?
(i.e. LVM on iscsi / iser or any other storage that works on raw images only)

Backups obviously don't work / are disabled due to this:
http://pve.proxmox.com/pipermail/pve-devel/2015-July/015984.html

With qcow2 snapshots work on NFS, but can be much slower than iscsi
(at least in my setup, ca. -40% or more on iops)

Did i get it right that there is neither backup nor snapshot possible for iothread=1 and iscsi ?
If so, are there any plans to change this in near future ?