Proxmox 5.1 2 Node HA Cluster

[Haider]

Hello all,

I am new to proxmox community. I have 2 proxmox clusters with 2 nodes each setup with ceph storage in same local network with Public IPs.

Recently one node on one of the cluster died but the secondary node on that cluster did not bring up the VMs.

I have done all the research and I am unable to find a way to have HA for a 2 node cluster. I want to be able to have 2 node HA cluster if one node dies the other one still brings up VMs.

Please advise.

 

[Haider]

root@hosting:~# pvecm status

Quorum information

------------------

Date: Wed Mar 7 15:55:22 2018

Quorum provider: corosync_votequorum

Nodes: 2

Node ID: 0x00000001

Ring ID: 1/9180

Quorate: Yes


Votequorum information

----------------------

Expected votes: 2

Highest expected: 2

Total votes: 2

Quorum: 2

Flags: Quorate


Membership information

----------------------

Nodeid Votes Name

0x00000001 1 192.168.1.116 (local)

0x00000002 1 192.168.1.117

 

[Haider]

Last night when I node failed I checked the other node it said something about waiting for Quorum. I had to manually power cycle 1st node bring cluster and VMs back up.

 

[Klaus Steinberger]

Sure , that is expected:
Expected votes: 2

So you need at least 2 votes for quorum.

For HA you _need_ at least 3 nodes.

My personal opinion: put all 4 nodes into _one_ cluster.
If you like to run this two separate clusters, then you have two possibilities:

-> best solution : add a additional 3rd node to both clusters

-> tricky solution:

put up a virtual proxmox server on both clusters and make it the 3rd node for the other cluster:

cluster1: node1 + node2 + virtualnode1 (running on cluster2)
cluster2: node3 + node4 + virtualnode2 (running on cluster1)

It should be obvious that the virtualnode is just for quorom and should not run VM's. It should also be obvious why virtual node for cluster 1 must run on cluster 2 and not on cluster 1 (and vice versa)

Also did you spread the CEPH over all 4 nodes or do you run separate CEPH's on both clusters or is it external?
If you run separate CEPH's on both clusters, this would be also a very weak setup, CEPH needs also at least 3 hosts (at least 3 OSD Hosts and 3 MON's, MON and OSD can be on same host).

 

[Haider]

Hey Klaus,

Thanks for detailed explanation and setup. I think I would go with your suggestion of putting all 4 nodes in 1 cluster.

Both clusters have separate CEPH although its same local network.

What would be best way to destroy cluster 1 and merge it to cluster 2. I already have VMs that were on cluster 2 backed up and restored to cluter 2 but not powered on.

 

[Haider]

Will this guide or node removal and adding to cluster 2 work for my setup.

https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node

 

[Klaus Steinberger]

Uhh, the remaining node will probably loose quorum

If the 2nd cluster is biug enough to run all vm's of both clusters I would move all machines (probably with a short downtime of the VM)
to the 2nd cluster, then reinstall both nodes from 1st cluster (its easier than cleaning up the cluster setup after removal), and add them the normal way.

 

[Haider]

I do want to do that but problem is I am remote and DC is in another state and I don't entrust installation and configuration part to locals.

I have read "pvecm expected 1" could help me bypass the quorum problem. Is that true ?

I would love to do it all remotely without losing my Public IPs if its not possible let me know so I start planning my trip to DC

 

[Haider]

I successfully reinstalled Proxmox on cluster 1 nodes and added them to cluster 2.

Some commands that helped just for reference after adding node to cluster

Ceph should be installed prior to joining the cluster

sgdisk -Z /dev/sdb (For Ceph Disk)
pveceph createosd /dev/sdb


After adding the node I could not see anything from slave nodes like Summary, Network and disk information so I had to run this command

pvecm updatecerts

Thanks for all the help Klaus