Server crash with full of "Memory cgroup out of memory"

[Linyu]

Hi forum,
I am suffering server crashes on some servers of my system.
These servers are running with few of LXC VM and KVM VM.
Suddendly the server crash without any sign, with the screen shows full of "Memory cgroup of of memory", this thing seems killed all things running on server.
The GUI is lost connection and bash is not useful at all, I must restart the server by force.
But these server are only installed a HDD in 7200RPM, the I/O delay is about 10-20 at normal time, I know it's quite low for a server running some many VM, but I want to know why it happen?


The last crash was when I am cloning a VM, and then it just happended. I don't know what to do because it will happen again and again, does it matter with bad disk I/O and heavy swap space usage?(KSM is enabled too)

 

[mplssilva]

good morning,
i have a similar problem (random restart) of some hosts using Proxmox based on issues - corosync / libknet:

https://forum.proxmox.com/threads/pve-5-4-11-corosync-3-x-major-issues.56124/page-12
- Solved in last update Proxmox 6.1-3 > .

What's the version of your Proxmox?
pveversion -v

- Using htop, the memory was made high buffer usage and using a lot of swap (even after proxmox 6.1-3).

- I saw a similar problem with asterisk, and bypass using crontab:

9 1-23/3 * * * sync; echo 1 > /proc/sys/vm/drop_caches

 

[Linyu]

good morning,
i have a similar problem (random restart) of some hosts using Proxmox based on issues - corosync / libknet:

https://forum.proxmox.com/threads/pve-5-4-11-corosync-3-x-major-issues.56124/page-12
- Solved in last update Proxmox 6.1-3 > .

What's the version of your Proxmox?
pveversion -v

- Using htop, the memory was made high buffer usage and using a lot of swap (even after proxmox 6.1-3).

- I saw a similar problem with asterisk, and bypass using crontab:

9 1-23/3 * * * sync; echo 1 > /proc/sys/vm/drop_caches

I am not using corosync but it still happends.
Using Zram now. the load average seems low down, I will check this later.

root@node-xz:~# htop



root@node-xz:~# pveversion -v

proxmox-ve: 6.0-2 (running kernel: 5.0.21-1-pve)
pve-manager: 6.0-6 (running version: 6.0-6/c71f879f)
pve-kernel-5.0: 6.0-7
pve-kernel-helper: 6.0-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.11-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-7
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-64
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1

 

[Giovanni]

I am not using corosync but it still happends.
Using Zram now. the load average seems low down, I will check this later.

root@node-xz:~# htop
View attachment 13776


root@node-xz:~# pveversion -v

proxmox-ve: 6.0-2 (running kernel: 5.0.21-1-pve)
pve-manager: 6.0-6 (running version: 6.0-6/c71f879f)
pve-kernel-5.0: 6.0-7
pve-kernel-helper: 6.0-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.11-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-7
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-64
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-5
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1

Were you able to fix this?

I'm seeing these "Memory cgroup out of memory" for Plex LXC container.