[SOLVED] Separate Cluster Network question?

[Spirog]

Separate Cluster Network​

When creating a cluster without any parameters, the corosync cluster network is generally shared with the web interface and the VMs' network. Depending on your setup, even storage traffic may get sent over the same network. It’s recommended to change that, as corosync is a time-critical, real-time application.

Setting Up a New Network​

First, you have to set up a new network interface. It should be on a physically separate network. Ensure that your network fulfills the cluster network requirements.

Separate On Cluster Creation​

This is possible via the linkX parameters of the pvecm create command, used for creating a new cluster.
If you have set up an additional NIC with a static address on 10.10.10.1/25, and want to send and receive all cluster communication over this interface, you would execute:
pvecm create test --link0 10.10.10.1
To check if everything is working properly, execute:
systemctl status corosync
Afterwards, proceed as described above to add nodes with a separated cluster network.

hello, I was reading and testing setting up Cluster.

I have a server and testing nested ( 3 VM's with 3 IP addresses) each has PVE installed.
the main server is hosted in a datacenter.
example IP 21.xx.xx.10 this has PVE installed

then I created 3 VM'swith PVE installed each has it's own IP address

node1 21.xx.xx.11
node2 21.xx.xx.12
node3 21.xx.xx.13

does this mean this setup is a Separate Cluster Network.

or because the 3 IP's are from the same gateway, and the server is connected to one 10gbit network - it does not qualify as Separate Cluster Network.

or is this for home use network configuration only ?

im not understanding this well, if anyone can help me understand better

thanks and sorry if it is a noob Question still trying to learn

thanks so much I appreciate your replies.

Kind Regards,
Spiro

 

[dylanw]

Hi, this doesn't look like a separate network. We regard a separate cluster network as one which sits on its own set of physical network interfaces, so that only corosync communication packets are passing through it.

In your case, if the three VMs have the IP address 21.x.x.x (e.g. on vmbr0) and all VM traffic uses this network (including corosync), it is a shared network, and corosync is susceptible to latency/load on that link.
Seeing as your instance is virtualised, an example of a separated network would be to assign each of the Proxmox VE VMs an additional virtual network interface from the VM's hardware panel (these interfaces should all be on the same bridge). Then, in the VMs, assign these new interfaces (e.g. ens19) an IP address within the same subnet as each other, e.g. 10.0.0.X/24. Finally, use this network as the 'link0' network when setting up the cluster. In this case, the VMs can continue using 21.x.x.x (vmbr0 on ens18) for other traffic, but corosync will have a dedicated network (ens19), which is independent and isolated from the load of regular node traffic.

Of course, being virtualised, the above example doesn't actually add redundancy unless a dedicated interface is genuinely added as the virtual interface, but it helps in understanding what we mean by and recommend in a separate cluster network.

 

[Spirog]

but it helps in understanding what we mean by and recommend in a separate cluster network.

Hello and Thanks SO MUCH @dylanw for this explanation.
makes more sense now.

Totally appreciate your help

Kind Regards,
Spiro