PVE6 slab cache grows until VMs start to crash
- Thread starter aflott
- Start date
small update:
on the virtual pve node finaly the corosync process was killed by oom, so i had to reboot:
overview of the growing slab for a week:
corresponding uptime:
you may find the attached pdf with all performance data collected from this node over a week.
on the virtual pve node finaly the corosync process was killed by oom, so i had to reboot:
Bash:
grep oom-kill /var/log/messages | tail -n 1
Oct 1 09:50:53 vspve4999 kernel: [419571.923228] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/corosync.service,task=corosync,pid=1286,uid=0overview of the growing slab for a week:
corresponding uptime:
you may find the attached pdf with all performance data collected from this node over a week.
The virtual pve-node was running with the mainline-kernel over the weekend?
That would indicate that the issue is probably in the upstream kernel - and not introduced by Ubuntu's or our patchset.
any chance you could try running the node with an older PVE-kernel (e.g. pve-kernel-5.0.12-1-pve)?
Thanks for reporting back!
That would indicate that the issue is probably in the upstream kernel - and not introduced by Ubuntu's or our patchset.
any chance you could try running the node with an older PVE-kernel (e.g. pve-kernel-5.0.12-1-pve)?
Thanks for reporting back!
yes, the node run with the mainline-kernel over the weekend.
the latest step today was to revert to the latest pve-kernel (5.0.21-6) which is a bit newer than the previous pve-kernel (5.0.21-2).
tldr. the changelog if the new kernel would address this issue?
oh, and i could not attach the pdf...too large.
you may find it here:
http://share.kohly.de/index.php/s/6sRi365ory7tEfA
the latest step today was to revert to the latest pve-kernel (5.0.21-6) which is a bit newer than the previous pve-kernel (5.0.21-2).
tldr. the changelog if the new kernel would address this issue?
oh, and i could not attach the pdf...too large.
you may find it here:
http://share.kohly.de/index.php/s/6sRi365ory7tEfA
hmm.. - one other user, who experiences this, reported that the issue was not present after booting 5.0.21-6 initially (which made me hope it might be fixed in that version)
Thanks for reporting back!
Thanks for reporting back!
Another user, who experiences the issue shared via user-list that potentially installed ipmi related packages could be a contributing factor (and from the report you shared I'd guess you're also using cmk/some other monitoring solution):
https://pve.proxmox.com/pipermail/pve-user/2019-October/171063.html
I hope this helps!
https://pve.proxmox.com/pipermail/pve-user/2019-October/171063.html
I hope this helps!
thanx for your research!
yes, we are using cmk for monitoring, and yes, we use ipmi to monitor the hardware.
so we will see...
yes, we are using cmk for monitoring, and yes, we use ipmi to monitor the hardware.
Bash:
root@vspve4999:~# aptitude search ipmi | grep ^i
i freeipmi - GNU implementation of the IPMI protocol
i A freeipmi-bmc-watchdog - GNU implementation of the IPMI protocol - BMC watchdog
i A freeipmi-common - GNU implementation of the IPMI protocol - common files
i A freeipmi-ipmidetect - GNU IPMI - IPMI node detection tool
i freeipmi-tools - GNU implementation of the IPMI protocol - tools
i ipmitool - utility for IPMI control with kernel driver or LAN interface (daemon)
i A libfreeipmi17 - GNU IPMI - libraries
i A libipmiconsole2 - GNU IPMI - Serial-over-Lan library
i A libipmidetect0 - GNU IPMI - IPMI node detection library
root@vspve4999:~# aptitude remove freeipmi freeipmi-bmc-watchdog freeipmi-common freeipmi-ipmidetect freeipmi-tools ipmitool libfreeipmi17 libipmiconsole2 libipmidetect0
<snip>
<snap>
root@vspve4999:~# reboot; exitso we will see...
The packages to install would be `bpfcc-tools` and `pve-headers`; then you'll have execsnoop-bpfcc.
Can just run this for a few minutes (as `execsnoop-bpfcc -tx` preferably). It will show what programs get executed.
hm - apart from the cmk-agent invocations and some regular timers from PVE I don't see too much going on on that host ...
Would it be possible to disable the cmk-agent service/socket on the host and see whether there's any effect? (you can continue monitoring with slabtop)
Thanks!
Would it be possible to disable the cmk-agent service/socket on the host and see whether there's any effect? (you can continue monitoring with slabtop)
Thanks!