[High Availability] Cause reboot all node?
- Thread starter kengrass
- Start date
if you want to enable HA, you need to ensure that your network is stable enough for corosync to not lose quorum:
https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_cluster_network
otherwise, it can happen that some or all nodes in the cluster get fenced:
https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#ha_manager_fencing
https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_cluster_network
otherwise, it can happen that some or all nodes in the cluster get fenced:
https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#ha_manager_fencing
1) So you mean this problem sure cause by when I turn on HA right ?
2) How to check logs of the "corosync" sir?
Sorry I'm totally new of proxmox
yes, this is HA doing its job. corosync notices nodes not being up/connected to eachother, and every node that is not part of the majority/quorum will "kill itself".
"journalctl -b -u corosync" will give you the log since bootup.
"journalctl -b -u corosync" will give you the log since bootup.
1) How do I know my cluster network is stable enough sir, I'm using Mellanox 40Gps for every nodes.
2) Do I need to config anything else or after beside turn on HA ( create group HA and turn on HA for every VM ) ?
Do you use just one network card for the cluster network? In that case the latencies may be the reason why corosync is losing the connection.
From the documentation [1]:
Maybe add a physical second network that is slow (1 GBit/s) and is completely reserved for corosync. See the chapter "Separate Cluster Network" in [1] for this.
[1] https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network
What a terrible design it is to mix the control plane and the data plane. It's our pain point too: disable HA and never use it!!!
I recommend that Proxmox learn from OpenStack and separate the cluster controller from the VM/computing node. Controller plane failures should not impact computing nodes. You can't 100% guarantee this will not happen, and you can't keep the network always stable; for example, a DDoS to one of your VMs will lead to the Proxmox host conntrack table being full or slow responses, then the control plane of the cluster is dead!!! Then all nodes reboot; that's a disaster and what's a bad design!!! If Proxmox doesn't fix this, it can't become a real enterprise solution.
I recommend that Proxmox learn from OpenStack and separate the cluster controller from the VM/computing node. Controller plane failures should not impact computing nodes. You can't 100% guarantee this will not happen, and you can't keep the network always stable; for example, a DDoS to one of your VMs will lead to the Proxmox host conntrack table being full or slow responses, then the control plane of the cluster is dead!!! Then all nodes reboot; that's a disaster and what's a bad design!!! If Proxmox doesn't fix this, it can't become a real enterprise solution.
Last edited:
Why? The main reason people run into issues isn't the layout of the UI but that they don't have a dedicated cluster network for corosync. And this they will need anyhow because otherwise you wouldn't be able to communicate the cluster configuration to the compute nodes.
I don't see a design or architecture issue if people are to lazy to read the documentation.
On the other hand the "mix" allows to use every node in a cluster as control node, so that you have the same managment features even if your mainly used node for controlling the cluster gets broken. You just need to enter another hostname in your browser or ssh client.
Nonsense. If you have (like recommended in the documentation) a dedicated cluster network just for corosync and additional use other networks as redundancy/failover networks for corosync this is rather unlikely. And I doubt that this is much different with Openstack (although I don't have any experience with it), since their documentation also states that they need a managment network on any node: https://docs.openstack.org/project-...26.1/targethosts.html#configuring-the-network
A Ddos to one of your VMs shouldn't cause any issue on the remainder of the cluster since the network connection to the vm doesn't go via the corosync network.
Unfortunately, a dedicated network cannot solve the problem I described, because the kernel itself crashed during the attack. No amount of network design helps in that scenario. OpenStack’s cluster control and compute functions are tightly coupled; it doesn't have a flawed design where a control plane failure triggers a reboot of the compute nodes.
And? As long as there are enough nodes to have a vote the cluster will still continue to work. So even if one host is victim to a ddos attack the other nodes still communicate with each other over the corosync network(s) and the cluster will continue to work.
But if you really think that this should be changed the bugzilla at https://bugzilla.proxmox.com is a better place than the forum since there is no guaranteed that developers will read everything here. The bugzilla however is closely monitored by them so at least one of them will read your suggestion.
Last edited:
That is generally not the case - this is often times a symptom of partial visibility (one node can see the other but not vice-versa). This can often happen when using e.g. LACP bonds.