Cluster with different number of links

[r7Y2]

I want to set up a cluster with 5 nodes, however only three have two nics, the other nodes have only one.

When I add an interface and the corresponding ring addresses to the three nodes in corosync.conf, the daemon throws the error Configuration is not valid: parse error in config: Not all nodes have the same number of links

So my question is: is a cluster with "unequal" nodes even possible and how should the corosync.conf look like?

 

[fba]

All nodes need access to all corosync rings. Do the single interface nodes have ip addresses for both corosync rings?
But aside of that: It is contradictory by itself to have a redundancy option not all members of a cluster support. So either get more interfaces or leave out the redundancy - with all the drawbacks.
Aside from that: The single interface statement is just for corosync or for all traffic of the server?

 

[r7Y2]

But aside of that: It is contradictory by itself to have a redundancy option not all members of a cluster support. So either get more interfaces or leave out the redundancy - with all the drawbacks.

Unfortunately, for the "remote" nodes, there will be no real redundancy, since they will be located in an adjacent building that is only connected by a single pair of fibers. Hence even if I dedicate another port, there will still be multiple single points of failure (fibers themselves, switches on both ends etc.). Thus I don't see the benefit of using more than one nic for corosync on the "remote" nodes.

All nodes need access to all corosync rings. Do the single interface nodes have ip addresses for both corosync rings?

I could configure IPs for the second ring on the single nic nodes, but packets can't be routed, since the second ring on the two nic nodes is on a physically separate network.
Corosync will run in a permanently degraded state on these nodes, but maybe that's the solution. I I'll try and report.

 

[UdoB]

Just use a simple USB-NIC for that second ring. This (personal view: USB-devices in general) is usually not recommended, but a second ring using sub-optimal devices is probably better than none.

Of course both connections should (hopefully) be stable under all circumstances; to check the current status use something like this:

~# corosync-cfgtool  -n
Local node ID 6, transport knet
nodeid: 2 reachable
   LINK: 0 udp (10.3.16.7->10.3.16.9) enabled connected mtu: 1397
   LINK: 1 udp (10.11.16.7->10.11.16.9) enabled connected mtu: 1397
   ...

And make sure the switch of the second ring is an independent one - it needs to be available when the "main" switch dies.

 

[r7Y2]

And make sure the switch of the second ring is an independent one - it needs to be available when the "main" switch dies.

We probably wrote our replies at the same time, hence you missed my explanation regarding that issue. But as I said in my second post, the remote site doesn't have an independent second physical link, hence no benefit in adding a second network that still has a single paint of failure...

 

[fba]

@r7Y2
Is it required for the remote nodes to be part of the same cluster? It might be an option to have a three-node main cluster and two stand-alone nodes in addition, depends on your setup and usecase of course.

 

[Johannes S]

Or split it up in two clusters (one three-node/one two-node) and add a qdevice to the two-node cluster. This could even be a VM in the three-node cluster, since the fiber connection should ensure a fast-enough connection:

Unlike corosync itself, a QDevice connects to the cluster over TCP/IP. The daemon can also run outside the LAN of the cluster and isn’t limited to the low latencies requirements of corosync.

https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

 

[VictorSTS]

Although I would setup two clusters, if you really want one cluster just setup corosync links in vlans and place said vlans on the available physical links on each host. Doesn't make sense for those "remote" nodes as it won't provide any real benefit, but won't hurt and at least the local nodes will have proper redundancy. If you ever get better redundancy towards the "remote" nodes you'll only need to move the vlans to the appropriate nic.

 

[r7Y2]

Or split it up in two clusters (one three-node/one two-node) and add a qdevice to the two-node cluster.

I initially thought about your and @fba 's idea as well, as an alternative solution. The main reason for my one cluster idea though is convenience and redundancy. I want to be able to do maintenance on one of the remote hosts (during production) without degrading the cluster to a state where I wouldn't consider it fit for production aka only one "real" node running.

 

[Johannes S]

Imho a five-node-cluster which is less relieable than the alternative (smaller clusters or single-nodes) is worse than no cluster at all. For maintenance you could also do a remote-migrate between different single-nodes or clusters with the Datacenter-Manager. With pve-zsync you could also replicate vms from one cluster to another one (you still need to manually launch them though)