high memory usage during backup

[cyp]

Hello,

I have a high memory usage during backup on two of my servers (other are fines).
Sometimes it's lead to VM crash

This is standalone install with soft raid (i know it's not recommended), one use ZFS and the other ext4.

Backup log

INFO: transferred 17179 MB in 36 seconds (477 MB/s)
INFO: archive file size: 3.92GB
INFO: delete old backup '/var/lib/vz/dump/vzdump-qemu-104-2017_07_29-03_04_34.vma.lzo'
INFO: Finished Backup of VM 104 (00:00:37)
INFO: Starting Backup of VM 105 (qemu)
INFO: status = stopped
INFO: update VM 105: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: qm-105
INFO: include disk 'virtio0' 'local:105/vm-105-disk-1.qcow2' 8G
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-105-2017_08_01-03_05_10.vma.lzo'
INFO: starting kvm to execute backup task
kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory
ERROR: Backup of VM 105 failed - start failed: command '/usr/bin/kvm -id 105 -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=205a6221-d6e2-4ea3-a0b9-e6d45c15953b' -name ase-mono01 -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/105.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k fr -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:47185d13f4a' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2' -drive 'file=/var/lib/vz/images/105/vm-105-disk-1.qcow2,if=none,id=drive-virtio0,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:00:00:51:f1:b9,netdev=net0,bus=pci.0,addr=0x12,id=net0' -S' failed: exit code 1
INFO: Backup job finished with errors
TASK ERROR: job errors

State of memory after a failed backug wich lead to vm crash.

root@proxmox:~# qm start 105
kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory
start failed: command '/usr/bin/kvm -id 105 -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=205a6221-d6e2-4ea3-a0b9-e6d45c15953b' -name ase-mono01 -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/105.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k fr -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:47185d13f4a' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2' -drive 'file=/var/lib/vz/images/105/vm-105-disk-1.qcow2,if=none,id=drive-virtio0,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:00:00:51:f1:b9,netdev=net0,bus=pci.0,addr=0x12,id=net0'' failed: exit code 1
root@proxmox:~# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       100 qm-101               running    3072              32.00 5857     
       102 qm-102               running    6144              20.00 3644     
       103 qm-103               running    4096              16.00 7214     
       104 qm-104               running    1024              16.00 4820     
       105 qm-105               stopped    1024               8.00 0         
root@proxmox:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          31859       30451        1214         123         192         986
Swap:             0           0           0
root@proxmox:~# qm stop 104
root@proxmox:~# qm start 104
root@proxmox:~# qm start 105
root@proxmox:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          31859       19748       11916         123         194       11689
Swap:             0           0           0

On Munin Graph the memory go high as soon as backup start

Any advices?

 

[Alwin]

ZFS uses RAM to cache writes, if you didn't already, you may try to limit the amount of ram that zfs takes.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_zfs_on_linux -> section 3.8.6. Limit ZFS Memory Usage

 

[cyp]

Thank for the reply, I will try that on the server using ZFS (it's the most problematic one, the other one just go down one time after upgrade but seems more stable since this crash).