Fencing and "isolation response"

[Pierre-Yves]

Hello
I was wondering how fencing works in detail.
I've never experienced real problems with Proxmox fencing thanks to the watchdog mechanism.
Nevertheless how does it works exactly ?

For instance : a cluster with 3 nodes running in HA, VM100 is running on host1.
Host1 stop responding on the corosync network but the VMs working on it continues to run (they're on another network and runs normally).
To my mind, in that case, the HA cluster will move VM100 on host 2 or host 3 but the VM100 is still running.
In some others platforms, there is a control on the locks of the virtual hard drive of the VM to prevent that kind of problems.

What about Proxmox ?

 

[danielb]

Proxmox will self-fence the isolated host : host1 will be forcibly rebooted, so VM100 is for sure not running anymore. While host1 is rebooting, VM100 will be relocated to either host2 or host3

 

[Pierre-Yves]

Thanks for your answer Daniel !
Rebooting the host is a drastical way to solve the problem.
I've never experienced such a thing by myself.
Where did you obtain that kind of information ?

 

[danielb]

In the doc, but also already tried ;-)

 

[Pierre-Yves]

But if the corosync network if offline, how the host can know he has to reboot ?

 

[fabian]

each node with HA active has to regularly write a timestamp to pmxcfs (/etc/pve). this will fail if a node is not part of the quorate partition of the cluster. the watchdog on each node monitors this timestamp, if it is not updated for a certain amount of time, the watchdog expires and the node is self-fenced. the CRM master (running on one node on the quorate partition) will monitor the timestamps as well, notice that one is expired, and will redistribute the HA resources belonging to that node accordingly. on the next boot of the fenced node, it will first sync up with the rest of the cluster, then it will see that all the resources went elsewhere and not be able to start a VM running on another node. the CRM will notice the node coming back up, and depending on HA settings, distribute HA resources to it again.

 

[Pierre-Yves]

Roohhhh
Thanks !!!