Proxmox restarts itself

[morphxyz]

one of our proxmox keeps restarting "randomly".
Especially when setting up new VMs. (on a network attached storage)

it's a cluster of 3. version 8.1.3

syslog just shows the reboot but no explanation.

what's your recommendation on how to figure what is causing these reboots?

Any help appreciated.

 

[morphxyz]

used journalctl --follow with no luck

The crash happened again - no new entries were made

 

[Darkk]

I would check your server's RAM. I've had this happened to two of my servers a few years ago till I was able to trace it to bad RAM. Took forever to figure out why as nothing reported in syslogs.

 

[morphxyz]

alright memtest it is then, I'll report back

 

[morphxyz]

Memory check is OK
CPU check OK
Disk check OK
NIC check OK
stresstest also OK
system log also kinda ok, except the reboots.

The issue persists.
System runs fine in idle.
Reboots only happen during stressful situations on that specific proxmox node.
Stresstest with a usb booted system runs through without errors.

So I guess it is related to proxmox but what do I know..

 

[sb-jw]

If they are enterprise servers, it could possibly be that a threshold value for the power supplies is triggered. If you have a cluster with HA, it could be that if the network load is too high, the Corosync traffic is lost and the watchdog comes into play.

You would just have to describe your setup in a little more detail so that we can get an idea of it here.

 

[morphxyz]

Those are 3 servers hosted by Hetzner.

They are connected with 10Gbit/s over a single Switch.
They all have a dedicated 1Gbit/s Internet connection, which is NOT connected to that switch. This connection is pretty much closed.
I only open port 8006 in case I can't access the proxmox from LAN.

Network load is low. Nothing's running on this cluster yet except a win22 DC and some other small things not reachable from the internet.

I did create a cluster. The machine having the issue is the "main"-node, which I used to join with the other two machines.

Just found some errors in dmesg. Are they related?

Attached: syslog before and after crash (reboot at 11:26:16)
attached: corosync.conf
attached: errors found in dmesg | grep -iE "error|failed"

 

[sb-jw]

You use OVS, right?

Dec 18 11:21:35 hetzner-pmx01 kernel: x86/split lock detection: #AC: CPU 0/KVM/2409367 took a split_lock trap at address: 0xfffff8012551049f

Maybe that's your problem. Compare it with the other nodes or search here in the forum, the error message has already been discussed here several times.

 

[morphxyz]

interesting! the other 2 machines have the exact same NIC and setup with OVS. But they both do not have any split lock detection entries in syslog. Would you recommend switching back to Linux Bridges and see if that changes anything?