Full mesh as failover cluster network

[sherminator]

Hi there,

I'm building a new 3 node PVE cluster. I already have a full mesh prepared for live migrations, Is it possible to use this full mesh as a failover network for the cluster itself?
In the WebUI at "Create Cluster" it doesn't seem to be possible to add both mesh links.

Thanks and greets!

 

[sterzy]

In the WebUI at "Create Cluster" it doesn't seem to be possible to add both mesh links.

What do you mean by "both" links? Corosync doesn't really care how exactly you set up your Full Mesh network. When creating the cluster you can just specify additional Links and set the IP address of the node on that link from the drop-down.

If you already created the cluster, you have to manually add the links to the Corosync config. See the manual for more info [1].

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[sherminator]

What do you mean by "both" links? Corosync doesn't really care how exactly you set up your Full Mesh network. When creating the cluster you can just specify additional Links and set the IP address of the node on that link from the drop-down.

If you already created the cluster, you have to manually add the links to the Corosync config. See the manual for more info [1].

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

Thanks for your quick reply!
I didn't create the cluster yet. Full mesh (at least in my case) means that there is one link to "node 2" and one link to "node 3", both with the same ip address, like this:

# Connected to Node2 (.51)
auto ens18
iface ens18 inet static
        address  10.15.15.50
        netmask  255.255.255.0
        up ip route add 10.15.15.51/32 dev ens18
        down ip route del 10.15.15.51/32

# Connected to Node3 (.52)
auto ens19
iface ens19 inet static
        address  10.15.15.50
        netmask  255.255.255.0
        up ip route add 10.15.15.52/32 dev ens19
        down ip route del 10.15.15.52/32

 

[sterzy]

I didn't create the cluster yet. Full mesh (at least in my case) means that there is one link to "node 2" and one link to "node 3", both with the same ip address, like this:

Yes so when you set up the cluster under Datacenter > Cluster > Create Cluster add an additional link and set the IP Address 10.15.15.50 from the drop-down. When joining the other nodes, the GUI will ask you for the new nodes IP addresses on all links, there you then select 10.15.15.52 or 10.15.15.51 depending on the node.

If the GUI does not pick up on the IP address correctly, make sure that they are configured correctly. If they are and it still doesn't work, you can always manually add a link later on as stated by our documentation [1].

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[sherminator]

and set the IP Address 10.15.15.50 from the drop-down.

sorry, I didn't get it yet. ens18 is a physical connection to node 2. Node 3 can't communicate with node 1 via this connection.
Do I overthink this? Should I just set the first mesh interface as the failover link for the cluster?

edit: ah, at the end only the IP address matters for the cluster, not the physical link, correct? So it doesn't matter which link I choose.

 

[sterzy]

edit: ah, at the end only the IP address matters for the cluster, not the physical link, correct? So it doesn't matter which link I choose.

Kind of, since you set the routes when the interfaces come up anyway, Corosync doesn't need to know how to communicate with a specific node as long as it can be routed to. Hope that makes sense