Snapshot backup forces a stop of the VM

[Greggy]

Hi,

I recently noticed that in the morning my VM's are all down. When looking at the logs of the backups, it seems that the snapshot backup is shutting down the VM's.
I'm currently running the latest Proxmox version:
root@proxmox:~# uname -a
Linux proxmox 4.4.35-1-pve #1 SMP Thu Dec 22 14:58:39 CET 2016 x86_64 GNU/Linux

From the backup log:
INFO: starting new backup job: vzdump 100 101 102 103 104 105 --storage Backup --mode snapshot --mailto ***@***.*** --quiet 1 --compress gzip --mailnotification always

INFO: Starting Backup of VM 104 (qemu)
INFO: status = stopped => The VM is running, not sure why it says it is stopped
INFO: update VM 104: -lock backup
INFO: backup mode: stop => mode should be snapshot
INFO: ionice priority: 7

I've checked also /etc/cron.d/vzdump to be sure there's no difference between the GUI and the configuration:
14 0 * * *root vzdump 100 101 102 103 104 105 --compress gzip --storage Backup --mailto ***@***.*** --quiet 1 --mode snapshot --mailnotification always

Does someone have an idea what is going on here?

Thanks a lot,
Greggy

 

[Greggy]

After disabling the backups, the next morning the VM's were down again. When looking at dmesg I could see the system ran out of memory and killed the VM's. So, it has nothing to do with the backups.

 

[wolfgang]

Do you use ZFS?
If yes have limit the memory?

 

[Greggy]

No, I'm using btrfs. I have 3 VM's with maximum 4GB of RAM and 1 with 1GB max RAM. That makes 13GB out of the available 16GB. At a given moment it start to kill the one VM after the other one.

[Wed Jan 4 20:18:55 2017] Out of memory: Kill process 13605 (kvm) score 173 or sacrifice child

[Wed Jan 4 20:18:55 2017] Killed process 13605 (kvm) total-vm:5238904kB, anon-rss:4228660kB, file-rss:11148kB

...

[Wed Jan 4 20:36:20 2017] Out of memory: Kill process 18170 (kvm) score 173 or sacrifice child

[Wed Jan 4 20:36:20 2017] Killed process 18170 (kvm) total-vm:5222492kB, anon-rss:4219516kB, file-rss:10924kB

...

[Wed Jan 4 20:58:21 2017] Out of memory: Kill process 17884 (kvm) score 170 or sacrifice child

[Wed Jan 4 20:58:21 2017] Killed process 17884 (kvm) total-vm:5226588kB, anon-rss:4151848kB, file-rss:10964kB


I found that BackupPc started at 20H00, so probably that one is causing the out of memory problem.

 

[wolfgang]

I do not use btrfs, so I'm not an expert on this,
but could it be that btrfs makes a operation and eat up your memory?

 

[Greggy]

I'm also not an expert, but I don't think so. Btrfs doesn't use a memory cache like zfs. I've disabled backuppc and now the VM's stayed up and running the next morning.

 

[fabian]

https://forum.proxmox.com/threads/p...ocess-8543-kvm-score-or-sacrifice-child.31569 <- updated kernel will follow soon.