Proxmox 4.4.5 kernel: Out of memory: Kill process 8543 (kvm) score or sacrifice child

udo said:
Hi,
how looks "cat /proc/sys/vm/swappiness" on the effected systems? Perhaps 0 instead of 1?

Udo
Click to expand...
I've had the OOM killer happen with swapiness set to 0, 1, 10 and 30

So far increasing the min_free_kbytes has prevented OOM events, also currently have swappiness set to 10.
The only thing using lots of RAM on my system is cache about 11GB when I experienced OOM events.
Only have 16GB RAM on these machines.
 
My host had running kernel 4.4.21-71 before upgrade and encounter OOM kill problem.

Other agency (no problem), supermicro motherboard too (no same models) :
1st : 64 GB, kernel 4.4.35-76
2nd : 128 GB, kernel 4.4.35-77

I had increasing the min_free_kbytes value this morning. The serial killer OOM has not yet hit :)
I will see tomorrow.

I will put again 4.4.21-71 kernel.
 
I reformatted one of my two affected servers (both Dell PowerEdge 2950-III systems, one with 28GB RAM, one with 16GB RAM) to not have any ZFS data pools whatsoever.
The non-ZFS server now survives the nightly backups.
The ZFS server still kills *both* VMs running on it. Just before the OOM killer kicks in, there's still >4GB free according to PVE's RRD graphs!
So far, backups seem to be what usually triggers the OOM problem.

Again, problem did not exist prior to the recent kernel upgrade... and since then, I've reduced the amount of memory allocated to the VMs so they never consume more than 50% of system RAM!
 
whitewater said:
Hi Udo, 60 :
Code: 
root@mtp-prox02:~# cat /proc/sys/vm/swappiness
60

I had done this on several proxmox host. All 60.
Click to expand...

swapiness was 60 too
min_free_kbytes was around 16MB (it was not set, so it was autocalculated to this value)

I have changed values respectively to 1 and 256MB, waiting for next backups to see if it changes anything
 
My VM victim was killed by OOM serial killer this afternoon, on 2nd proxmox host. Kernel 4.4.35-77 and min_free_kbytes had 262144 value. 32 GB RAM (same 1st hardware).

I can now run kernel 4.4.21-71 on one server to get all stable.

I will run the same VM from a backup, as test, with your test kernel Fabian.
 
Last edited:
Since yesterday evening :
1st server : Fabian test kernel with a VM test, a backup / restore of the VM victim.
No killed at this moment. 8 GB ram used of 32 GB.

Only this VM is running. The VM was killed even if if running alone, with 4.4.35-77 kernel.

2nd server : kernel 4.4.21-71, 5 VMs, 17 GB ram used of 32 GB.
Backup are OK, no VM killed at this moment.

Let see in time if it stable.

moarph, do you talking about 4.4.35-77 kernel ?
 
Last edited:
Hi.
I am new to proxmox and have the same problem. I would like to try the test kernel but dont know how to do it. I have already downloaded the files to "tmp" but need the commands to install them and use the kernel on boot.
Thanks.
 
Hi Jorge, to install this kernel, do :
Code: 
dpkg -i pve-kernel-4.4.35-2-pve_4.4.35-78~test1_amd64.deb
Reboot and login.
Code: 
uname -a
must return 4.4.35-2-pve
 
Same here.
>= 25% RAM available, swap nearly empty.

It's not only killing the kvm-process:
Jan 5 00:16:32 serverA kernel: [607936.489702] Out of memory: Kill process 3660 (node /root/dash) score 17 or sacrifice child
Jan 5 01:17:13 serverA kernel: [ 3484.604788] Out of memory: Kill process 17484 (kvm) score 202 or sacrifice child
Jan 5 03:20:17 serverA kernel: [10868.264749] Out of memory: Kill process 9641 (node /root/dash) score 16 or sacrifice child

Jan 5 11:41:57 serverB kernel: [62710.635363] Out of memory: Kill process 30604 (pveproxy worker) score 15 or sacrifice child
Jan 5 11:41:59 serverB kernel: [62712.748791] Out of memory: Kill process 30622 (pveproxy worker) score 15 or sacrifice child

I just installed the test-kernel on both machines - no problems so far.
I'll keep watching the syslog for "Out of memory".
 
whitewater said:
moarph, do you talking about 4.4.35-77 kernel ?
Click to expand...

Yes I had problems with the stock kernel. with the following one: "pve-kernel-4.4.35-2-pve: 4.4.35-78~test1" backup works like it always used to work. My biggest problem was I blamed everything else but Proxmox because I put a new RAID card in and upgraded my backup-server.
 
For me, "more simple". No hardware upgrade.
Problems occurs after updating proxmox, with new kernel. So, i think about maybie a kernel "bug" and see this topic.

The main one is to have found the solution.
OOM serial killer did not reoffend at this moment. Pratically one day. Keep our fingers crossed :)
 
Similar issues here since upgrading on 1st Jan - which I presume relate to the updated updated the kernel.

Crash ocurs shortly after a backup runs.

Linux proxmox 4.4.35-1-pve #1 SMP Thu Dec 22 14:58:39 CET 2016 x86_64 GNU/Linux

I have done the following to see what happens:

cat > /etc/sysctl.d/90-my.conf <<DELIM
vm.swappiness = 1
vm.min_free_kbytes = 262144
DELIM

sysctl --system

That way you can delete the file and revert to standard as required

If that works I'll try the updated kernel.

B, Rgds
John
 
@john: Only adding the swapping and the min_free_kbytes didin't do the job for me. The system performed slightly better but kvm still crashed. After the kernel update i deleted the swapiness and the min_free_kbytes and it still works very well.
 
moarph said:
@john: Only adding the swapping and the min_free_kbytes didin't do the job for me. The system performed slightly better but kvm still crashed. After the kernel update i deleted the swapiness and the min_free_kbytes and it still works very well.
Click to expand...

Ok - thanks. I'll give it a whirl tonight as is and see what happens (always helps to test these things) and I can give the kernel a try tomorrow

B. Rgds
John
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back