3 node HA cluster Proxmox with Ceph

[sanojs]

Hi All,

I have installed 3 node HA cluster with ceph storage. And one VM is installed on a node and tested the HA.

We had stopped 1 HA node (proxmox1) and the installed VM has moved to other online HA node(For example: from proxmox1 to proxmox2). But when we tried to powered off both pve node (proxmox1 and proxmox2), the VM is not switched to the third node (proxmox3).

Is there any other way we can test the 3 node cluster HA with both nodes brought down with installed VM swictch to third node.


Thanks,

 

[tom]

tried to powered off both pve node (proxmox1 and proxmox2), the VM is not switched to the third node (proxmox3).

If you loose quorum (2 out of 3), the HA cluster do not work anymore.

You need to make sure that you do not loose the quorum.

Testing and learning how the HA stack is working:

=> HA Simulator https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_ha_simulator

 

[sanojs]

Hi Tom,

Suppose if you loose the quorum and VM went to down state.

Is there anyway to bring up the VM with one node?

We are using ceph storage. Please assist

Thanks

 

[tom]

Is there anyway to bring up the VM with one node?

Fix at least one missing node so you have quorum again.

 

[sanojs]

Hi,

I have run this command
#pvecm expected 1
And the vm is visible, but the ceph storage is not visible and not working, so the VM start failed timeout error it shows. Kindly help me to provide steps to recover the server in this case.

 

[tom]

How can it happen that you loose 2 nodes from your HA cluster without the ability to repair them?

I just want to understand your setup so that we can assist.

Without knowing details, I suggest you restore your VM from last backup (after you fixed your cluster).

 

[sanojs]

Hello,
I am on testing the 3 node proxmos HA enviornment with CEPH has storage which is configured on /dev/sdb and OSD is creating for this sdb for all three nodes. So i need to know how can we run the VM in single node if the two other nodes in the cluster fails.

 

[tom]

So i need to know how can we run the VM in single node if the two other nodes in the cluster fails.

You cannot. You need at least 2 nodes up and running (in a 3 node setup).

If you want to survive 2 node failures, you need at minimum a 5 node cluster.

 

[ermanishchawla]

Dear @sanojs

Proxmox cluster and Ceph cluster are two independent cluster

Proxmox cluster can even work with Single node by manually calling "pvecm expected 1"

Ceph cluster requires minimum 3 nodes, If you look at your crush map using "ceph osd tree" You will see that osd are identified under "Host" and by default crush rule ensures PG are placed in different host and now if you look at the configuration of ceph pool using command
"ceph osd dump | grep pool" You will see that it is either SIze 3 min_size 2 ( Recommended) or if you have configured something else it would show

That line means for every object data will be placed in 3 OSD and Minimum 2 are required
and your crush rules says every Object will be placed such that 3 OSD are from 3 different Hosts
thus it simply says "You cannot withstand 2 failures" in a 3 node setup with default configuration