How to create a Proxmox cluster when the nodes do not belong to the same local network

[greenorange]

Hi,

I purchased 3 VPS from a hosting provider and installed Proxmox on each of them.
Unfortunately they do not have internal networking option to connect the 3 together. Hence I'm creating this thread to seek alternative approach.

My Setup:
- Each of the 3 proxmox nodes are accessible over the internet.
- Each of the 3 proxmox nodes have a public ip-address.
- These 3 nodes are not part of a local network.


What I'm trying to accomplish:
- node1 has vm1, vm2.
- node2 has vm3, vm4.
- node3 has vm5, vm6
- I would like vm1 vm2 vm3 vm4 vm5 vm6 - all to be in the same local network
(Eg: 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 10.0.0.6)


Is it even possible to do it since the 3 nodes are "not" doesn't belong to the same local network?


Note:
Yes I'm fine with providing public access to the 3 nodes. These are just 3 test instances I've created just for practice purposes.


Please assist.

 

[B-C]

Seems it might be possible using vxlan possibly depending on the connectivity fabric between the nodes.

So additional issue you have to look at is if you attempt to cluster them now you run into some other issues with VMs Existing.
I did cluster after having VMs and not recommended but is possible.

Depending on the provider you should be able to relocate the nodes to the same network.
If not - then see if you can setup a vxlan on the SDN.

This gets tricky depending on MTU on the switching between sites I found.

I'd play with it and see what is possible - but I'm no expert!

 

[KciNicK]

create a virtual network on each node without attaching a physical network card, create 1 vm on each node (could be a tiny debian, minimal install), add two virtual network cards to each vm, one to the physical network, and one to the other, assign the network 10.0.0.0 to the no physical card, create a tunnel between the vms (gre, ipip, geneve, etc), bridge the tunnel with the 10.0.0.0 card, and that's it, you have a private network between all the nodes, i use to do that on OVH before the vrack was available, i was using vmware, but for the case, it's the same thing.

 

[sw-omit]

create a virtual network on each node without attaching a physical network card, create 1 vm on each node (could be a tiny debian, minimal install), add two virtual network cards to each vm, one to the physical network, and one to the other, assign the network 10.0.0.0 to the no physical card, create a tunnel between the vms (gre, ipip, geneve, etc), bridge the tunnel with the 10.0.0.0 card, and that's it, you have a private network between all the nodes, i use to do that on OVH before the vrack was available, i was using vmware, but for the case, it's the same thing.

The problem with that (and many other options) is delay:
You need to have a stable and less then 5 (or with just 3 nodes max 10) ms connection between all the nodes, for the cluster itself to be stable.
Since you're going over the internet, even with dedicated fibre connections, delays might sometimes shift, especially if other traffic is going over the same internet-line, which could break things in a cluster setup. Again, could work just normally (see the more reading too), but not really supported, and unlike vmware, there is no "master" node, so communication between all nodes is much more important.

Source: https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

The Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably. While on setups with a small node count a network with higher latencies may work, this is not guaranteed and gets rather unlikely with more than three nodes and latencies above around 10 ms.

More reading: https://forum.proxmox.com/threads/high-latency-clusters.141098

 

[KciNicK]

that's another problem, my comment was to create a private network between the nodes, there's no need to create a cluster, according to the op post, i don't think he needs one, even when he mentions the word "cluster" in the thread title.

 

[sw-omit]

Ah yeah, missed that part, the title was indeed misleading.

If you do it just to connect the VM's without making a proxmox-cluster, it might work like described (but still probably not optimal)

Alternatively if the main concern is to (safely/easily) reach each-other, then setup up a virtual router with site-to-site vpn options is probably easier.
Yes they will have to be in different IP-ranges, but the router will work out who should go where.
If you also make the ranges smaller then a /24, for example /26 or even /29 with the ranges being next to eachother, if you do decide to combine the nodes to one network in the end, all you'd have to do is change the subnet on all and the default gateway on some of the devices, and all the connections will keep working.

 

[KciNicK]

yes, vpn is another option, but i prefer the tunnels as it has L2 forward too, you install it and forget about it, you even can setup a dhcp server on on node and get ip from another, also no need to split networks, all depends on your needs.