some nodes suddenly go offline in a cluster of five nodes

[helj15798]

After fresh installation of Proxmox VE (7.0-2) for five servers, I created a cluster on one node pub1 and added the rest four nodes into this cluster.

Everything works well after the cluster creation and I can see all five nodes online when I use the web interface of any nodes.

But after about one hour (I did not do anything during this period),
- I can only see yard3 and yard4 online when I log into the web interface of yard3.
- I can only see pub1, yard5 and yard6 online when I log into the web interface of pub1.



After rebooting all the five servers,
- I can only see pub1 itself when I log into the web interface of pub1.
- when I log into yard3, I see pub1 offline and yard345 with a question mark.
- the web interface lags and it even takes a long time to refresh.



Interestingly, after shutdown of pub1, the accesses to yard456 are restored and everything works well.
Once I turn on pub1, the web interface gets laggy again.



In addition, I saw the following INFO on all the servers.



I have also tried to re-install PVE on all the five servers. But similar thing happens again...
Anyone know what happed and how do I solve this issue?