Best practices for host + cluster organization, two DCs

[cctgteddy]

Hi,
I'm working on migrating our environment off vmWare. The relevant part of my environment is below

DC1 - prod
Proxmox Environment
2x Dell R660 - Dual Xeon Silver 4410Y

vSphere Environment
3x Dell R740 - Dual Xeon Bronze 3106
3x Dell R740 - Dual Xeon Silver 4214

DC2 - off site services + test/dev
vSphere Environment
2x Dell R340 servers


The vmWare hosts are all accessible through a single vCentre server appliance, in both DCs. They are all operating as standalone hosts, no clusters set up, no vSAN, no vSwitch, etc.

The 2 Proxmox servers are configured together as a 2-node cluster.

To start moving these 8 other Dell servers into Proxmox should I:
1. 3 separate clusters with matching host configs?
2. Join them all to the existing cluster?
3. Create a second cluster for DC2, but join the other 6 in DC1 to the existing cluster?


I really prefer option 2 if it gives me a single point to manage the VMs. I'd also like to start leveraging better automation and HA options that I've never taken adantage of in vmware.

TIA

 

[MarkusKo]

I'm not an expert but first of all you should add a qdevice to your existing pve cluster, you need at least 3 devices in the cluster to achieve a valid quorum.

Don't add servers in different DC's to a single cluster (except if you are absolutely sure the connection between both DC's is very reliable and low latency even when migration VM's between data centers).

If the latency is too high the 2 servers in DC2 will randomly reboot because they loose the connection to the cluster (especially when there is other data transfer in the background like a migration of a VM or copying data between DC's - latency will rise ....)

I would recommend to create one cluster in DC1 (add a qdevice, you need an odd number)
If needed create a second cluster in DC2 (again with a qdevice)
You could also virtualize the qdevice for DC1 as a VM in DC2 and vice versa - a qdevice has not the same latency requirements like a cluster node with corosync.

To migrate VM's between the clusters you could use Proxmox DCM but it's in alpha state right now ...

If you go the cluster route make sure you add redundant dedicated network links for cluster traffic (corosync).

Because you have different CPU's in your cluster you should not use "host" as CPU in your VM config but something like "Skylake Server" (the highest architecture that works on your older Xeon Bronze CPU's) otherwise you could get problems / errors when live migrating VM's between different hosts.

This is just for a pve cluster, if you intend to create a ceph cluster there are even more things to consider.

And don't forget to read through the offical docs / requirements

Hope this helps

 

[cctgteddy]

Thanks. Was thinking it might be safest to have two DCs. I'll look at adding the qdevice to my 2-node environment.
My DC1 will be 8 nodes (or more) so not sure it's needed there once I start adding more nodes.

As mentioned my vmWare environment isn't using any clustered features, I just like having a single point of management (vCenter) all hosts. (There are several older hosts too that I didn't include which may be retired). Moving to 1 per DC is fine. Proxmox DCM sounds slick.

I've very very rarely used vMotion between my two locations but I like the option.

 

[alexskysilk]

To start moving these 8 other Dell servers into Proxmox should I:
1. 3 separate clusters with matching host configs?
2. Join them all to the existing cluster?
3. Create a second cluster for DC2, but join the other 6 in DC1 to the existing cluster?

Based on your original criteria- why bother clustering anything at all? Since it appears all you're really after is a single pane of glass- leave them all as standalone servers and use PDM for the control plane.

Clustering makes sense when you intend to use them as resource providers for the same applications/vms.

 

[MarkusKo]

A corosync cluster needs an odd number of votes, if you have an even number of nodes then you should add a qdevice or another node for quorum votes.

If you don't want the overhead of a cluster (dedicated redundant NIC's / LAN subnets for corosync) then you could just run all nodes standalone, downside is you have to login to every node for management, you only see the resources of that node you are logged in and stuff like SDN has to be configured for every node.

PDM does not offer many features yet, you can migrate VM's between nodes but you can't really mange the nodes themself from PDM. That could change in the future ...

 

[cctgteddy]

Based on your original criteria- why bother clustering anything at all? Since it appears all you're really after is a single pane of glass- leave them all as standalone servers and use PDM for the control plane.

Clustering makes sense when you intend to use them as resource providers for the same applications/vms.

This is good feedback - "single pane of glass" is indeed my primary goal, however I'd like the flexibility to utilize them as a combined resource now/in the future. My R740s in my primary DC are similar enough I think I'll make a single cluster + the qdevice to do an odd number. Thanks @MarkusKo for the info on this - I misunderstood that it just needed 3+, not an odd number!

In DC2 I'll create a second cluster with the 2 identical hosts + qdevice.

Having two management consoles instead of 10 is still better than individually managing hosts.

There are a few older hosts which I'll deploy as standalone nodes.

Thanks for the feedback everyone!

 

[Johannes S]

I misunderstood that it just needed 3+, not an odd number!

It's actually a bit more complicated, tldnr: An odd number is recommended because it changes the behaviour in case of a node outage:

We support QDevices for clusters with an even number of nodes and recommend it for 2 node clusters, if they should provide higher availability. For clusters with an odd node count, we currently discourage the use of QDevices. The reason for this is the difference in the votes which the QDevice provides for each cluster type. Even numbered clusters get a single additional vote, which only increases availability, because if the QDevice itself fails, you are in the same position as with no QDevice at all.
On the other hand, with an odd numbered cluster size, the QDevice provides (N-1) votes — where N corresponds to the cluster node count. This alternative behavior makes sense; if it had only one additional vote, the cluster could get into a split-brain situation. This algorithm allows for all nodes but one (and naturally the QDevice itself) to fail. However, there are two drawbacks to this:

• If the QNet daemon itself fails, no other node may fail or the cluster immediately loses quorum. For example, in a cluster with 15 nodes, 7 could fail before the cluster becomes inquorate. But, if a QDevice is configured here and it itself fails, no single node of the 15 may fail. The QDevice acts almost as a single point of failure in this case.
• The fact that all but one node plus QDevice may fail sounds promising at first, but this may result in a mass recovery of HA services, which could overload the single remaining node. Furthermore, a Ceph server will stop providing services if only ((N-1)/2) nodes or less remain online.

If you understand the drawbacks and implications, you can decide yourself if you want to use this technology in an odd numbered cluster setup.
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support