H.A. freezed situation

[samontetro]

Hi,

I've set up a 3 nodes cluster with CEPH (1 disk/node) and High availability. I'm in the testing step now (not in prod) and I spend some time to broke the config and check how proxmox recovers.
I fall in a freezed situation where:
- I have a separate network for CEPH
- my linux VM is running on node3, all filesystems on CEPH storage, exporting a NFS storage to a linux client.
- I launch a write from the NFS client (2GB file to take enougth time) on the NFS storage and I break the CEPH network on node3 (unplug the CEPH network cable)

Node 3 as no quorum for CEPH now, the VM freeze.
I try to migrate manualy the VM on node 2 but I get:
qmigrate:100:root@pam:' still active, waiting
for ever.

How can I configure proxmox HA to get out such a situation ? Something like watchdog in the VM ?

Thanks

Patrick

 

[t.lamprecht]

Hi,

the VM is frozen in the sense that it does not respond anymore or in the sense that the HA manager has as state "freezed" for it when executing `ha-manager status`?

I suspect the former.

What settings do you use for the ceph pool, especially for replica and size?

I guess its 2/1? If so change that to 3/2, meaning that you have 3 replicas and at least 2 must be written that Ceph tells the client it successfully saved the data.
Else you will get into split brains. As said, just a guess, but the symptoms you describe support it.

 

[t.lamprecht]

- I have a separate network for CEPH

Oh, the VM just could hnag because Ceph cannot complete its IO transactions.

To address such issues you need redundancy on the network level as well, i.e. bonding of the Ceph network.

 

[samontetro]

Hi Thomas,

Yes, when I say frozen, I mean that the VM do not answer anymore to ssh and NFS access on the exported partition is not working on the Client. I've not tested connecting to the VM from the node itself. But the VM state is running on the web interface may be because it is using a different network interface.
CEPH is on eth8
Proxmox admin is on vmbr0 connected to eth0
VM is bridged on vmbr9 connected eth9

My CEPH is set up with 3/2 of course to avoid split brain (and I'll add 3 new disks in a few week to reach 2 osd/node)

Plugging again the Ceph network cable on the node, proxmox recovers and finishes to execute my migration request but I would like that VM to be moved (stopped/killed and restarted on an other node) where Ceph has quorum in case of network failure.

 

[samontetro]

Oh, the VM just could hnag because Ceph cannot complete its IO transactions.

To address such issues you need redundancy on the network level as well, i.e. bonding of the Ceph network.

Redundancy at this level is not possible as for security reasons the third node is not in the same building and I've only one physical network support between the two buildings.
Isn't it possible to kill the VM in this situation ?

 

[t.lamprecht]

My CEPH is set up with 3/2 of course to avoid split brain (and I'll add 3 new disks in a few week to reach 2 osd/node)

Plugging again the Ceph network cable on the node, proxmox recovers and finishes to execute my migration request but I would like that VM to be moved (stopped/killed and restarted on an other node) where Ceph has quorum in case of network failure.

Ok, then my second post above should address this.

You told ceph that it needs to write at least two copies of any object before returning from an IO operation.
Now with just one connection between the nodes Ceph network a disruption of said network hinders ceph on writing the second copy and thus it doesn't return from the IO operation until the network works again.
This leads to a VM with processes hanging in IO state.

 

[t.lamprecht]

Redundancy at this level is not possible as for security reasons the third node is not in the same building and I've only one physical network support between the two buildings.
Isn't it possible to kill the VM in this situation ?

Yes, the VM Stop command should work. Over the WebUI or the CLI with:

qm stop VMID -timeout 0

The "timeout 0" tells the command to not wait long for a cleaner stop and kills the process on the host.

 

[samontetro]

I've simulated again a communication failure on the CEPH network, unplugin the network cable on node3 where the VM was running.
OK, I can stop the VM from the graphic interface. May be last try has failed as first I had requested à migration of the VM which was waiting for ever.
When the VM is stopped, I can migrate it on node2 and restart it.

Is there a way to automate this to provide self recovery of the HA service is case of local network failure?

When the VM freeze on the I/O, ha-manager do not seams to see the problem:

root@proxmost2:/etc/pve/qemu-server# ha-manager status
quorum OK
master proxmost1 (active, Thu Mar 23 15:44:55 2017)
lrm proxmost1 (idle, Thu Mar 23 15:45:00 2017)
lrm proxmost2 (active, Thu Mar 23 15:44:52 2017)
lrm proxmost3 (active, Thu Mar 23 15:44:53 2017)
service vm:100 (proxmost3, started)
​

When plugin back the Ceph network cable, Ceph recovers safely with no obvious problem.

Patrick

 

[samontetro]

I finaly found a solution to have some resilience when the CEPH network fails on one node: I've implemented a small daemon testing (with ping) the availability of the two other nodes on this CEPH dedicated network every minutes. After 4 failures to reach all the other nodes, the current node is halted and this leads proxmox HA to restart the VM on an available node. In case of major network problem all the cluster will stop. That's OK.

Initialy I was stopping the VM before halting the node with the suggested command "qm stop VMID -timeout 0" but this disable restarting "automagically" the VM on an other node (and I think it is a correct behavior).

Is there a safer way to stop the freezed VM before issuing the halt command on the node and allowing its automatic migration ?