backup fails

[dfedorov]

Hi! Not performed backup for the virtual machine. Memory is enough. What could be the problem?

Spoiler: task log

INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: update VM 100: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: PrW2k8R2
INFO: include disk 'virtio0' 'local-zfs:vm-100-disk-1' 570G
INFO: stopping vm
INFO: creating archive '/mnt/pvebk/dump/vzdump-qemu-100-2017_03_03-03_00_02.vma'
INFO: starting kvm to execute backup task
kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory
INFO: restarting vm
INFO: start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=8b3ab887-7562-4dd2-aae9-570157a8144c' -name PrW2k8R2 -smp '10,sockets=1,cores=10,maxcpus=10' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/100.vnc,x509,password -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,enforce' -m 40000 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cbe0e8aaba' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=86:EF:03:10:34:13,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
INFO: kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory
command 'qm start 100 --skiplock' failed: exit code 255
ERROR: Backup of VM 100 failed - start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=8b3ab887-7562-4dd2-aae9-570157a8144c' -name PrW2k8R2 -smp '10,sockets=1,cores=10,maxcpus=10' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/100.vnc,x509,password -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,enforce' -m 40000 -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cbe0e8aaba' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=86:EF:03:10:34:13,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard' -S' failed: exit code 1
INFO: Backup job finished with errors
TASK ERROR: job errors

Spoiler: free -m

total used free shared buffers cached
Mem: 48190 24681 23508 44 2716 104
-/+ buffers/cache: 21861 26329
Swap: 8191 29 8162

 

[fabian]

you only have 26329+8162 memory left in total (that is, after dropping caches and including all swap), but your VM wants 40000.

 

[dfedorov]

you only have 26329+8162 memory left in total (that is, after dropping caches and including all swap), but your VM wants 40000.

but the VM is stopped during backup

 

[fabian]

no, for VMs we backup using qemu's I/O layer, so the VM needs to either be running, or if it isn't (or you are using "stop" mode), vzdump will start the VM in a "non-running" mode to allow this access.

 

[dfedorov]

no, for VMs we backup using qemu's I/O layer, so the VM needs to either be running, or if it isn't (or you are using "stop" mode), vzdump will start the VM in a "non-running" mode to allow this access.

those. to backup VMs with 40GB of RAM I need 80GB on the host system? even if I use the stop mode?

 

[fabian]

those. to backup VMs with 40GB of RAM I need 80GB on the host system? even if I use the stop mode?

backing up a VM has the same requirements as starting it: all the disks and devices need to be available, and you need enough RAM. you don't need twice as much RAM - I don't know why you think that?

 

[dfedorov]

backing up a VM has the same requirements as starting it: all the disks and devices need to be available, and you need enough RAM. you don't need twice as much RAM - I don't know why you think that?

I have 48Gb RAM on the server. I have allocated 40GB for VM. I use "stop" mode for backup. and I get an error: cannot set up guest memory 'pc.ram': Cannot allocate memory

 

[dcsapak]

I have 48Gb RAM on the server.

yes but "only" 26 GB free

you need as much free memory as the vm needs

 

[dfedorov]

yes but "only" 26 GB free

you need as much free memory as the vm needs

We did not understand each other. I performed "free -m" when the VM was running. I have only one virtual machine on the server. "free -m" when a single virtual machine is stopped:

.             total       used       free     shared    buffers     cached
Mem:         48190       4663      43527         45          0         98
-/+ buffers/cache:       4564      43625
Swap:         8191         24       8167

 

[dcsapak]

ok, but at the time of the backup you had definitely not enough free memory (it also could not start it afterwards, according to the log)
so i would try again to (manually) backup under the same conditions and look for an application which uses the ram

 

[dfedorov]

ok, but at the time of the backup you had definitely not enough free memory (it also could not start it afterwards, according to the log)
so i would try again to (manually) backup under the same conditions and look for an application which uses the ram

ok, i will try.

 

[dfedorov]

ok, but at the time of the backup you had definitely not enough free memory (it also could not start it afterwards, according to the log)
so i would try again to (manually) backup under the same conditions and look for an application which uses the ram

Hi, I tried to start a backup manually. Please look at the screenshots. Processes sorted by memory usage.

 

[dcsapak]

i see a few things here:

on the "after" screenshot, you can see you have 9905/48190 MB allocated Memory -> you have less than the needed 40GB free

also i can see that you use zfs, which by default, can allocate up to 50% of your memory, and it can happen, that it does not release the memory fast enough needed for kvm.
you can limit the memory usage of zfs see https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage

i would recommend limiting your zfs usage and installing more memory or reducing the amount of memory for the vm