Snapshot backup kills KVM VM

[frantek]

Hi,

I've some hosts where a single KVM VM of N gets killed while snapshot backup and so the backup fails. For all other KVM VMs the backups work as expected. It is allays the same VM on each host which crashes while backup. The VMs are completely different - it is not the same VM on all hosts. The problem KVM VMs run Debian 6, 7 and 8 releases. One Debian 8 some time just stops during normal operation. Sometimes they survive the backup. The problem hosts run Proxmox 3.x.

How to debug this and what may be the reason for the crashes?

TIA

 

[chrisu]

Hello, since applying the latest 4.4 - Updates I have the same problem here. 6 of 8 VM are killed during backup. Two, for what reason ever survive.

I do not have any idea what is happening. The logs do not show anything pointing to a problem....only seeing that oom-killer is triggered during backup.
ksmtuned invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0

But there is a lot of memory available. Does anybody have a suggestion?

Thanks and greetings

Chris

 

[fabian]

see https://forum.proxmox.com/threads/p...ocess-8543-kvm-score-or-sacrifice-child.31569 for users experiencing similar issues (and a potential workaround)

 

[chrisu]

Hello Fabian,
after three days of testing I have discovered that according to your referenced posting both settings must be set to take any effect:
vm.swappiness = 1
vm.min_free_kbytes = 262144

Nevertheless tonight one VM was stopped by oom-killer:
Jan 7 05:50:00 pve1 kernel: [412142.839504] kvm invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0

Anything else to do to have all machines surviving the backup?

Thanks and best greetings

Chris

 

[chrisu]

Hi,

today two machines died during backup. Any further actions to taken?

Thank you and greetings
Christ

 

[fabian]

Hi,

today two machines died during backup. Any further actions to taken?

Thank you and greetings
Christ

did you install the new kernel and reboot?

 

[chrisu]

Hi,

not rebooted yet. But I have not seen that the new kernel addresses oom-killer errors.

I have scheduled the reboot for tonight. Lets see whats happening.

Greetings
Chris

 

[fabian]

Hi,

not rebooted yet. But I have not seen that the new kernel addresses oom-killer errors.

I have scheduled the reboot for tonight. Lets see whats happening.

Greetings
Chris

you have to reboot for the new kernel to run.. "uname -a" should report 4.4.35-2-pve, built on January 9th

 

[chrisu]

Hi Fabian,

tonight all VMs survived.

Greetings

 

[chrisu]

Hi,

a short update: Still no oom-killer issues with the new kernel. Seems that the problem is now solved. Thank you for your support.

Best greetings

Chris