Proxmox Cluster Networking question?

[killmasta93]

Hi,

i was wondering if someone could shed some light on what im trying to do. Currently have Site to site Openvpn working great no issue. what im trying to do on Site B i have a cluster on proxmox, the idea is that i want to take one of the servers to Site A, but the problem is that it has a different Ip from Site A, would it be possible to move that server to Site A and somehow keep the same IP and be able to communicate with the Site B with the site to site?

im trying to wrap my head around it not sure if i need to VLAN on the edge swich or how could i approach this?


https://imgur.com/a/ABi0aMy



Thank you

 

[wolfgang]

Hi,

Do you try to run a cluster over a VPN?
If yes this does not work.
The latency is too high and multicast does not work over VPN tunnels that easy.

 

[killmasta93]

Thanks for the reply, currently i have the cluster not though Site to site, the idea is to have only 1 node on the other side of the site to site VPN. as for the latency not sure would a problem, my issue is that node from site A has a different subnet from Site B which i was reading need VXLAN and OVS bridge. I put the OVS bridge working but not sure whats the next step to VXLAN

 

[wolfgang]

If your latency is higher than 7 ms the problems will start in the hole cluster not only on the one outside.
Do yourself a favor and make a long time benchmark to prove if you always under the 7ms.

 

[killmasta93]

Thanks for the reply, well your right the ms im getting is 20-28 max when you mean hole not sure what you mean?

 

[wolfgang]

max when you mean hole not sure what you mean?

Proxmox VE cluster uses a real-time filesystem to keep information and states.
If one node is always flapping it will disturb the cluster filesystem and you will not able to manage it.

 

[killmasta93]

would it be and issue if i throw the cluster for that node with the WAN ip? instead of the LAN?

 

[crainbramp]

As a CSP who runs Proxmox clouds for a living, I can tell you ... this is a BAD idea.

The problem is, that the cluster network is VERY sensitive to latency. This is why it's recommended, and really a necessity for production clusters, that your cluster traffic be separated out on it's own redundant bond by itself. We run 8 NICs per node to get the redundancy and latency that we need.

Once your latency gets over that threshold (7ms-10ms) the ENTIRE cluster will start having issues. Not just the "remote" node in this case. It can even get so bad that the cluster will start rebooting nodes, and if you're thinking about HA stretched across DCs ... just dont. The only customer we have doing this successfully does it between our Seattle Data Center and Amazon on a direct-connect line with <3ms latency, and buddy, they PAY for that connection.

As Always,
Crain

 

[wirdo02]

would it be and issue if i throw the cluster for that node with the WAN ip? instead of the LAN?

I've been struggling with the same issues you are right now.
After beeing in contact with the proxmox team I went with pve-zsync instead. It does require a manual action to turn on the VM on the other side though. I made a small webpage for my collegues to enable the VM in case our main location goes down.

Probably not the prettiest way to do it but at least it's stable this way.
Every 15 minutes a replication is being made to the offside location.

Also if I'm correct; looking at your graph you would have 3 servers in one location and 1 in the other.
If side B fails you have no quorum and will result in your entire cluster going on halt

 

[killmasta93]

yeah correct i ended up using the server on site B off the cluster and just use pve-zsync to backup the VMs