Questions regarding Corosync and Q-Device

[rothkraut]

Hi!

I am currently migrating several ESX-Cluster to PVE.
We have two data centers spread across two buildings. An even number of nodes is spread across those two data centers. (e.g. 3 nodes in one and 3 in the other one)

I installed a Q-Device in a third seperate room (and seperate builing on our campus). Works fine and as intended. We use two networks for Corosync. One main and one fallback.

We want to set up several PVE-Cluster within our data centers (one main, one DMZ, one "lagacy" etc.) For every cluster a Q-Device is needed.

Now i have two questions for my upcoming setup:

Can I use the heartbeat network for all my clusters?
Lets say for my main heartbeat ring I use a Network 10.20.30.0/24 and use the addresses 10.20.30.1 - 6 for one Cluster. Would it be a problem if i use 7 - 12 for the next one etc.

If so, is it possible to use the same Q-Device with the same IP for every Cluster? Or is it necessary to configure diffrent IPs within the network on that one q-device?

I don't want to set up a diffrent Corosync-VLAN for every cluster, but well if it is necessary...

Thanks in advance!

 

[lucavornheder]

Because Corosync is running in unicast mode, you can keep everything in one network. That said, it’s cleaner to have a separate network for each cluster. On the host that runs Corosync, I’d also install Proxmox as a single node and host the qDevice as a VM (or LXC).

 

[UdoB]

We use two networks for Corosync. One main and one fallback.

Great! Just make sure to use two independent physical connections. All VLAN in a single trunk will possibly fail at the same time...

Can I use the heartbeat network for all my clusters?

Sure. The limiting factor is (high) latency. And because that is often induced by congestion it should be separated from the high-traffic frontends and/or storage networks.

Disclaimer: I do not have actual multi-cluster experience...

 

[Johannes S]

An even number of nodes is spread across those two data centers. (e.g. 3 nodes in one and 3 in the other one)

If you happen do have at least three nodes in a cluster with an odd amount you shouldn't add a qdevice:

"We support QDevices for clusters with an even number of nodes and recommend it for 2 node clusters, if they should provide higher availability. For clusters with an odd node count, we currently discourage the use of QDevices. The reason for this is the difference in the votes which the QDevice provides for each cluster type. Even numbered clusters get a single additional vote, which only increases availability, because if the QDevice itself fails, you are in the same position as with no QDevice at all. "
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

Can I use the heartbeat network for all my clusters?
Lets say for my main heartbeat ring I use a Network 10.20.30.0/24 and use the addresses 10.20.30.1 - 6 for one Cluster. Would it be a problem if i use 7 - 12 for the next one etc.

I don't see a problem. I would avoid a setup, where you would assign the same IP in both clusters or connect from one network to the other. Corosync is quite latency-sensitive so I would remove any potential issues caused by routing stuff from host a to host b ( both in location 1 ) via host c ( in location 2 ).

If so, is it possible to use the same Q-Device with the same IP for every Cluster? Or is it necessary to configure diffrent IPs within the network on that one q-device?

again from the already linked wiki page:

Unlike corosync itself, a QDevice connects to the cluster over TCP/IP. The daemon can also run outside the LAN of the cluster and isn’t limited to the low latencies requirements of corosync.

So the qdevice can even have an IP on a completely different IP subnet. This is what I would do and give each cluster it's own dedicated IP subnet so it's always clear which nodes belong to which cluster and how they are connected

 

[rothkraut]

Thank you for your comprehensive answer.

If you happen do have at least three nodes in a cluster with an odd amount you shouldn't add a qdevice:

Well the 6 nodes I mentioned belong to the same cluster. So its one cluster split into two seperate buildings. And since both data centers have the same amount of Nodes I added a Q-Device in a third location to prevent a split brain situation in case of e.g. power failure in one data center.



Thank you very much your awnsers regarding the networkthemed questions. So i wil set it up the way i planned.

I know seperate corosync-networks for each cluster would be the best solution. But due to internal circumstances I would like to avoid that for now.

 

[rothkraut]

Great! Just make sure to use two independent physical connections. All VLAN in a single trunk will possibly fail at the same time...

The main corosync is connected to each node with one physical nic and the switchport is configured as a normal access port.
The backup corosync is connected to each node with a seperate 2x25G bond. This bond is used for storage and backup traffic.

Yeah I know high traffic interfaces, but with vmware we connected storage and backup with 2x10G and it was never fully needed. We just upgraded to 2x25G because we can.

 

[Johannes S]

Thank you for your comprehensive answer.


Well the 6 nodes I mentioned belong to the same cluster. So its one cluster split into two seperate buildings. And since both data centers have the same amount of Nodes I added a Q-Device in a third location to prevent a split brain situation in case of e.g. power failure in one data center.

That's a absolutely legit usecase for a qdevice then since we are now talking about a stretched cluster instead of two seperate ones
A setup like this is described on https://pve.proxmox.com/wiki/Stretch_Cluster (although it's motly about Ceph but the part on requirements etc. should cover other storages too.
I assume everything is connected in a low-latency-network e.g. dark fibre? If not I would seperate both clusters to avoid issues with non-sufficient network connections. Location C I would then use for a ProxmoxBackupServer and go without the qdevice.

I know seperate corosync-networks for each cluster would be the best solution. But due to internal circumstances I would like to avoid that for now.

I thought you have one (stretched) large cluster instead of two different ones? Both setups would work but change whether the qdevice is a good idea or not.

 

[rothkraut]

I thought you have one (stretched) large cluster instead of two different ones? Both setups would work but change whether the qdevice is a good idea or not.

Yeah my wording wasn't very good.


Currently I have one streched Cluster with a qdevice. This is our main cluster.

I want to set up several more streched clusters. (e.g. Testing, DMZ, Legacy etc.)
I want to set them up in the same manner.

But I would just like to use one qdevice (with the same IP) and the same corosync networks (main and fallback) for all my clusters.

 

[Johannes S]

Yeah my wording wasn't very good.


Currently I have one streched Cluster with a qdevice. This is our main cluster.

I want to set up several more streched clusters. (e.g. Testing, DMZ, Legacy etc.)
I want to set them up in the same manner.

But I would just like to use one qdevice (with the same IP) and the same corosync networks (main and fallback) for all my clusters.

Using the same qdevice is no problem as long your network setup ensures that each node can reach the qdevice. For example in my homelab my cluster has two corosync networks (main and fallback) but the qdevice is on a completely different subnet on a host outside of the cluster. The gateway router takes care, that the qdevice is reachable from all cluster nodes
I'm not sure I understand your intended setup for the corosync networks. Do you want to have them all on the same physical network? This is something I would avoid. Or do you want to use the same IP subnets although they live on different physical networks which are not connected to each other? This should be ok although I wouldn't do it to have a clearer assignment which IP belongs to which cluster.

 

[rothkraut]

Or do you want to use the same IP subnets

Yes I want several diffrent cluster to use the same IP subnets. But I do see your point. It would be better organized to have diffrent subnets for each cluster.
But since there isn't a technical reason to avoid this, I will go with this setup for now.

Thanks for your input everybody