hard proxmox 5.3 crash caused by HA

[Daniel Pfeilsticker]

Hi,
our setup:
two "big" proxmox 5.3.8 server with zfs-filesystem and HA configured for 3 VMs
one old server only as quorum-server.

Steps to reproduce:

1. As we have new switches we wanted just to plug network-cables from old to new switch (with some seconds of interruption).
2. So we first unplugged the third quorum-server, so the HA would not trigger any VM migrations.
3. unplug network of server-2 for 5 seconds
4. about 20 seconds later server-1 crashes hard an restarts

From server-1 syslog:

• Feb 22 13:13:23 proxmox-1 corosync[6597]: [MAIN ] Completed service synchronization, ready to provide service.
  • fine: both servers back in sync, seeing and knowing each other
• Next logline after hard crash:
  • Feb 22 13:15:18 proxmox-1 systemd[1]: Starting Flush Journal to Persistent Storage...

Lost some data by this hard crash so setup the same environment on three old servers (proxmox 5.3.2 from iso-image)

And here the same effect:
Looking at the monitor/shell of server-1 there is even no kernel-panic message on the screen before hard power down...

Steps to prevent:

• No problem when the third quorum-server stays plugged in.
• No crash if all/the HA-group and HA-configurations are deleted before.
  
  • Cluster and replications-jobs can stay active.
  • quorum-server can be unplugged as long as HA is deleted.

As there is no log at all and it is reproducible and a little bit unbelievable her the hand-made;-) video:
drive.google.com/file/d/1zxmtd-WyMO61FTP7Bq8R-jappPmWJaqM/view?usp=sharing

Daniel

 

[dcsapak]

• So we first unplugged the third quorum-server, so the HA would not trigger any VM migrations.

• unplug network of server-2 for 5 seconds

am i understanding you correctly that you unplugged 2/3 servers? if yes then this is exactly what ha is supposed to do? the one node still online (with ha services active), fences itself
because it no longer has quorum...

or do i misunderstand something?

 

[Daniel Pfeilsticker]

Yes:
server 3 unplugged before and unplugged all the time
server 2 unplugged for 20 seconds
server 1 plugged in with connection to the network for these 20 seconds
PS: the unplugged server 2 keeps running fine with all VMs online.

Server 1 decides to hard power off after server 2 is back and you see corosync to be back in sync between server 1 and 2.
No HA migrated or double-running VMs as without server 3 we don't have quorum all the time.

I might understand a planed/graceful shutdown of all VMs and after VMs shutdown the server.
But a hard power off with damaged databases and no hint in syslog?
And this after a short (in this case planed) network failure?

Daniel

PS: Im now offline until Thuesday, so have a nice Weekend

 

[dcsapak]

I might understand a planed/graceful shutdown of all VMs and after VMs shutdown the server.
But a hard power off with damaged databases and no hint in syslog?

check the journal as well, sometimes the message do not get synced to the disk

the idea is the following:

if a server loses quorum, we have to make sure that the server is no longer online, and this is done via self fencing
did you read the ha documentation already? https://pve.proxmox.com/wiki/High_Availability

what i do not understand is why you remove the quorum node from the network?

the reason of a third node is that you do *not* lose quorum because it provides an additional vote, so 1 server can be gone and the cluster still has quorum
if you now remove the quorum node from the network, no other node can fail without losing quorum

 

[Daniel Pfeilsticker]

Hi,

Yes i read https://pve.proxmox.com/wiki/High_Availability last year but did not care enough about "self fencing"...
I did expect a more graceful fencing, something like a regular shutdown...
Especially because we don't have a shared storage (NAS/SAN/drbd) that would cause a VM running twice from the same storage.

The stupid idea to unplug our third server-3 for a longer time:
a) cleanup and rearrange our third "server-room"
b) prevent HA failover/migration from server-1 to server-2 while one of them is unplugged only for some seconds

But indeed this case could happen in normal operation as well, for example if our glass fiber between our offices fails:
So we would have server-2 and -3 still connected, but server-1 alone in the first office:
server-2 and -3 will have quorum 2of3 and will start all HA-VMs running still on server-1 as well.
So if we have a reconnect server-1 must fence = terminate all double running VMs.

But it would be nice to fence more graceful, for example take the VM-network-interfaces down and trigger a shutdown instead of a hard stop...

Daniel

 

[dcsapak]

Especially because we don't have a shared storage (NAS/SAN/drbd) that would cause a VM running twice from the same storage.

why do you have ha active if you have no shared storage? that makes no sense...

b) prevent HA failover/migration from server-1 to server-2 while one of them is unplugged only for some seconds

you could do this with ha groups, have each vm in a ha group restricted to the node it is on, but again it makes no sense to have ha enabled in this situation

But indeed this case could happen in normal operation as well, for example if our glass fiber between our offices fails:
So we would have server-2 and -3 still connected, but server-1 alone in the first office:
server-2 and -3 will have quorum 2of3 and will start all HA-VMs running still on server-1 as well.
So if we have a reconnect server-1 must fence = terminate all double running VMs.

this is exactly the reason of the self fencing.. to prevent running vms twice... a regular shutdown is not enough because it can block and since between
the disconnected server there is no way to communicate, we have to agree to a time limit at which the watchdog triggers and the server self-fences

But it would be nice to fence more graceful, for example take the VM-network-interfaces down and trigger a shutdown instead of a hard stop...

as i said this is not enough since we have to guarantee that the whole node is off when the others start to 'steal' vms
only network is not enough because of other shared resources (e.g. storage)

 

[Daniel Pfeilsticker]

Hi,

HA works fine with simple replication if you are aware to loose all changes between last replication and HA-switch.

For us we have only DNS, DHCP and vpn configured as HA-VMs so we can live with loosing one hour of changes but can fix it from home via vpn...
And we still have Internet even if our offices are disconnected as in this case we have DNS- and DHCP-VM running twice = once in each office...

Yes, you are completely right that in most other use-cases the hard power off fencing is the right solution.

Daniel

 

[dcsapak]

HA works fine with simple replication if you are aware to loose all changes between last replication and HA-switch.

ok did not know you were using replication, this makes the setup more understandable

but still, if you want to (even temporarily) stop the relocation of vms, use ha-groups

you can have a ha-group for each host with the vms inside and both nodes in normal use, and if you want to prevent some vms to move, just remove the other node in the group temporarily