2-node cluster second host hard-resets on first host reboot

[Sprinterfreak]

TL;DR if one host is rebooted, the second hosts hard-resets. No logs in dmesg/messages.

I have a little setup running on ThinkCenters, with nvme formatted as lvm. This setup serves a few home applications and is mainly for educational purposes. A playground so to say. Both hosts have 16GB RAM and 8GB (default) swap on lv [2].
There is no automatic HA relocate configured. Both hosts serve some lxc and qemu guests.

From time to time, when I update the hosts for instance, I need to reboot just one node. I would then expect the node which I rebooted manually to reboot and join the cluster, then bring up its VM's again. What actually happens is, that the second node hard-resets as soon as it doesn't reach the first node anymore. Then if both hosts booted up again (both in state waiting for quorum) VM's start-up and the cluster becomes available. Both hosts memory usage did not exceed 20% when this happend, Swap usage was 0B out of 8GB. Since there is no HA action configured, I do not expect much io before the second node dies.

My first question is: Is this normal behavior in a 2-node cluster? (I guess: no)
The second question is how to gather useful information to track down the cause of the second node's kernel to just die and reboot.
Sadly there is nothing logged to /var/log/messages when this happens. It looks like it dies that hard, that it does not even try to flush anything to disk.

It was happening on at least 5.13.19-6-pve, 5.15.39-2-pve and current 5.15.83-1-pve.

Maybe related to:
[1] https://forum.proxmox.com/threads/hosts-rebooting-at-the-same-time.33068/ - no cause identified or
[2] https://forum.proxmox.com/threads/security-pve-can-be-crashed-by-user-on-demand.49343/ - zfs host crashes on swap usage

I guess in a production environment this would be quiet disastrous.

 

[Neobin]

Sounds like the normal fencing: [1].
Are you sure, you have nothing (maybe from previous testing) HA-related set up? My thought was/is, that in such a case like yours, the other node "only" loses quorum and the cluster gets read-only...

Edit: Of course, could also be something completely different.

Anyway: With only two nodes, a QDevice: [2] is highly recommended!

[1] https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#ha_manager_fencing
[2] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_corosync_external_vote_support

 

[Sprinterfreak]

Yes, you seem to be totally right. Fencing really kills the host. There was HA configured on a vm template...

Not only that solved, thank's for pointing out the slim qdevice. There is of course a 3rd node in progress but this really is the newly available(?) fix for that. If I remembered right, had seen a discussion on the qdevice years ago but had missed its final existence.

 

[Sprinterfreak]

My thought was/is, that in such a case like yours, the other node "only" loses quorum and the cluster gets read-only...

This is actually what I expected too. Thats why I migrated my kind of important vm's manually over to the second node before first node reboot. I expected the cluster to get read-only (no management input possible) but not to die completely. I also don't have shared ressources within the VM's, so did not see the case of split-brain. So resetting the last node was the worst thing what could possibly happen in this case.