Proxmox 4.4.5 kernel: Out of memory: Kill process 8543 (kvm) score or sacrifice child

e100

Renowned Member
Nov 6, 2010
1,248
33
68
Columbus, Ohio
ulbuilder.wordpress.com
Hi,
how looks "cat /proc/sys/vm/swappiness" on the effected systems? Perhaps 0 instead of 1?

Udo
I've had the OOM killer happen with swapiness set to 0, 1, 10 and 30

So far increasing the min_free_kbytes has prevented OOM events, also currently have swappiness set to 10.
The only thing using lots of RAM on my system is cache about 11GB when I experienced OOM events.
Only have 16GB RAM on these machines.
 

whitewater

Member
Nov 26, 2012
107
0
16
france
My host had running kernel 4.4.21-71 before upgrade and encounter OOM kill problem.

Other agency (no problem), supermicro motherboard too (no same models) :
1st : 64 GB, kernel 4.4.35-76
2nd : 128 GB, kernel 4.4.35-77

I had increasing the min_free_kbytes value this morning. The serial killer OOM has not yet hit :)
I will see tomorrow.

I will put again 4.4.21-71 kernel.
 

athompso

Member
Sep 13, 2013
127
8
18
I reformatted one of my two affected servers (both Dell PowerEdge 2950-III systems, one with 28GB RAM, one with 16GB RAM) to not have any ZFS data pools whatsoever.
The non-ZFS server now survives the nightly backups.
The ZFS server still kills *both* VMs running on it. Just before the OOM killer kicks in, there's still >4GB free according to PVE's RRD graphs!
So far, backups seem to be what usually triggers the OOM problem.

Again, problem did not exist prior to the recent kernel upgrade... and since then, I've reduced the amount of memory allocated to the VMs so they never consume more than 50% of system RAM!
 

opty

Member
Apr 5, 2012
20
0
21
France
Hi Udo, 60 :
Code:
root@mtp-prox02:~# cat /proc/sys/vm/swappiness
60

I had done this on several proxmox host. All 60.

swapiness was 60 too
min_free_kbytes was around 16MB (it was not set, so it was autocalculated to this value)

I have changed values respectively to 1 and 256MB, waiting for next backups to see if it changes anything
 

whitewater

Member
Nov 26, 2012
107
0
16
france
My VM victim was killed by OOM serial killer this afternoon, on 2nd proxmox host. Kernel 4.4.35-77 and min_free_kbytes had 262144 value. 32 GB RAM (same 1st hardware).

I can now run kernel 4.4.21-71 on one server to get all stable.

I will run the same VM from a backup, as test, with your test kernel Fabian.
 
Last edited:

moarph

Active Member
Dec 22, 2013
31
0
26
Same here...latest Kernel will kill my windows machines during backup...
 

whitewater

Member
Nov 26, 2012
107
0
16
france
Since yesterday evening :
1st server : Fabian test kernel with a VM test, a backup / restore of the VM victim.
No killed at this moment. 8 GB ram used of 32 GB.

Only this VM is running. The VM was killed even if if running alone, with 4.4.35-77 kernel.

2nd server : kernel 4.4.21-71, 5 VMs, 17 GB ram used of 32 GB.
Backup are OK, no VM killed at this moment.

Let see in time if it stable.

moarph, do you talking about 4.4.35-77 kernel ?
 
Last edited:
Oct 30, 2016
20
0
21
52
Hi.
I am new to proxmox and have the same problem. I would like to try the test kernel but dont know how to do it. I have already downloaded the files to "tmp" but need the commands to install them and use the kernel on boot.
Thanks.
 

whitewater

Member
Nov 26, 2012
107
0
16
france
Hi Jorge, to install this kernel, do :
Code:
dpkg -i pve-kernel-4.4.35-2-pve_4.4.35-78~test1_amd64.deb
Reboot and login.
Code:
uname -a
must return 4.4.35-2-pve
 

2beers

New Member
Jan 16, 2013
9
0
1
Same here.
>= 25% RAM available, swap nearly empty.

It's not only killing the kvm-process:
Jan 5 00:16:32 serverA kernel: [607936.489702] Out of memory: Kill process 3660 (node /root/dash) score 17 or sacrifice child
Jan 5 01:17:13 serverA kernel: [ 3484.604788] Out of memory: Kill process 17484 (kvm) score 202 or sacrifice child
Jan 5 03:20:17 serverA kernel: [10868.264749] Out of memory: Kill process 9641 (node /root/dash) score 16 or sacrifice child

Jan 5 11:41:57 serverB kernel: [62710.635363] Out of memory: Kill process 30604 (pveproxy worker) score 15 or sacrifice child
Jan 5 11:41:59 serverB kernel: [62712.748791] Out of memory: Kill process 30622 (pveproxy worker) score 15 or sacrifice child

I just installed the test-kernel on both machines - no problems so far.
I'll keep watching the syslog for "Out of memory".
 

moarph

Active Member
Dec 22, 2013
31
0
26
moarph, do you talking about 4.4.35-77 kernel ?

Yes I had problems with the stock kernel. with the following one: "pve-kernel-4.4.35-2-pve: 4.4.35-78~test1" backup works like it always used to work. My biggest problem was I blamed everything else but Proxmox because I put a new RAID card in and upgraded my backup-server.
 

whitewater

Member
Nov 26, 2012
107
0
16
france
For me, "more simple". No hardware upgrade.
Problems occurs after updating proxmox, with new kernel. So, i think about maybie a kernel "bug" and see this topic.

The main one is to have found the solution.
OOM serial killer did not reoffend at this moment. Pratically one day. Keep our fingers crossed :)
 

reetp

Active Member
Aug 19, 2013
49
3
28
Similar issues here since upgrading on 1st Jan - which I presume relate to the updated updated the kernel.

Crash ocurs shortly after a backup runs.

Linux proxmox 4.4.35-1-pve #1 SMP Thu Dec 22 14:58:39 CET 2016 x86_64 GNU/Linux

I have done the following to see what happens:

cat > /etc/sysctl.d/90-my.conf <<DELIM
vm.swappiness = 1
vm.min_free_kbytes = 262144
DELIM

sysctl --system

That way you can delete the file and revert to standard as required

If that works I'll try the updated kernel.

B, Rgds
John
 

moarph

Active Member
Dec 22, 2013
31
0
26
@john: Only adding the swapping and the min_free_kbytes didin't do the job for me. The system performed slightly better but kvm still crashed. After the kernel update i deleted the swapiness and the min_free_kbytes and it still works very well.
 

reetp

Active Member
Aug 19, 2013
49
3
28
@john: Only adding the swapping and the min_free_kbytes didin't do the job for me. The system performed slightly better but kvm still crashed. After the kernel update i deleted the swapiness and the min_free_kbytes and it still works very well.

Ok - thanks. I'll give it a whirl tonight as is and see what happens (always helps to test these things) and I can give the kernel a try tomorrow

B. Rgds
John
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!