Quorum dissolved HA VM shutdown

[Ninjix]

Is this expected behavior?

I just restarted two out of my four PVE 2.0 RC1 test nodes at the same time. The rgmanager reported the quorum dissolved as node-4 was shut down. The HA machine (VM 100) was shutdown by the cluster on node-01 even though that node remained online. Once node-3 completed its restart and the Quorum was regained, VM 100 was restarted on node-02. Any ideas on what may have caused this?

Before:

node-01	Online	VM 100 running
node-02	Online
node-03	Online
node-04	Online

During:

node-01	Online	VM 100 shutdown
node-02	Online
node-03	Restarting
node-04	Restarting

After:

node-01	Online
node-02	Online	VM 100 running
node-03	Online
node-04	Online

Here is a copy of the rgmanager log from node-01.

...
Mar 05 15:00:07 rgmanager [pvevm] VM 100 is running
Mar 05 15:00:08 rgmanager [pvevm] VM 100 is running
Mar 05 15:00:38 rgmanager [pvevm] VM 100 is running
Mar 05 15:00:49 rgmanager #1: Quorum Dissolved
Mar 05 15:00:51 rgmanager [pvevm] Task still active, waiting
Mar 05 15:00:52 rgmanager [pvevm] Task still active, waiting
Mar 05 15:02:33 rgmanager Quorum Regained
Mar 05 15:02:34 rgmanager State change: Local UP
Mar 05 15:02:34 rgmanager State change: node-02 UP
Mar 05 15:02:34 rgmanager Loading Service Data
Mar 05 15:02:36 rgmanager Skipping stop-before-start: overridden by administrator
Mar 05 15:02:36 rgmanager [pvevm] VM 100 is not running
Mar 05 15:02:40 rgmanager State change: node-03 UP
Mar 05 15:02:52 rgmanager Migration: pvevm:100 is running on 2
Mar 05 15:02:52 rgmanager [pvevm] VM 100 is not running
Mar 05 15:02:53 rgmanager [pvevm] VM 100 is not running
Mar 05 15:03:23 rgmanager State change: node-04 UP
Mar 05 15:03:23 rgmanager [pvevm] VM 100 is not running
EOF

 

[Ninjix]

I just repeated another test. This time I live migrated VM 100 to node-03. Then I restarted node-01 and node-02. VM 100 was again shutdown. It restated this time on node-04. Could something be off with an index value of my cluster?

 

[dietmar]

It restated this time on node-04.

Where do you se a restart? Looks more like an online migration (or is the VM really stopped)?

 

[dietmar]

Is this expected behavior?

I agree this is not optimal, but you should really avoid loosing quorum. Please can you try to specify a failover domain to test is behavior is better (see 'man rgmanager')

 

[dietmar]

Ok, here is the full explanation. When a cluster member determines that it is no longer in the cluster quorum, the service manager stops all services and waits for a new quorum to form. That happens on node1 (stop VM).

When the cluster get quorate rgmanager starts the service again (chooses any node if you do not specify a fail-over domain).

see 'man clurgmgrd'

 

[Ninjix]

Thank you, Dietmar. You explanation is consistent with my test results. I've also learned from the rgmanager docs that a quorum = (0.5 * n) + 1 which also explains why a shutdown of 2/4 nodes broke the quorum and the service manager stopped the VM on node1. Now that I understand how this works, I am planning to test HA with ordered fail-over domains. From there I going to setup some Zabbix monitoring rules for these conditions.