CPU use shoots up to 100% and Oom-killer is invoked when adding additional node to 3-node cluster.

[PVE-Daemon]

I'm attempting to create a 5 node cluster using Truenas as storage over iSCSI and as soon as I try and join a 4th and 5th node, the cpu jumps up to anywhere form 95-120% on each node and the oom-killer is invoked.. killing the corosync service. I've experimented to see what happens when the oom killer is adjusted, but if corosync isn't killed, something else is.... crashing the system. I've added more RAM and even reinstalled using a new box. To no avail.
Dmesg output


journalctl -u corosync.service output

 

[dcsapak]

how much memory do your nodes have? how much of that memory is assigned to vms? do you use zfs? (that can use up to 50% of your memory)

 

[PVE-Daemon]

I haven't spun up any vms yet. I'm not using zfs.

 

[dcsapak]

ok this is strange, since the error message above says it's killed with about ~360MiB of memory in use..

can you post the complete output of 'dmesg' and the journal from before the cluster join until the oom? (e.g. with 'journalctl --since=<since> --until=<until>')

 

[PVE-Daemon]

Sure.

 

[dcsapak]

did you do any tuning of sysctl or similar?
it seems that somehow the memory allocation fails, maybe due to memory fragmentation
(but if it is like you say and you don't have anything running yet on that node, i don't see how that could be)

i see in the logs that you don't run the newest version of the 5.15 kernel, maybe upgrading can help
(or alternatively, booting into the 5.13 kernel to test)

 

[PVE-Daemon]

I'm running the 5.15 kernel now, with the same issue.

 

[dcsapak]

and with an earlier 5.13 ? just trying to find some hint at what we're looking here? (maybe some kernel memory leak?)

 

[PVE-Daemon]

Same issue with 5.13