Wireguard and remote datacenter assembly (servers in diffrent phisical locations). WG + PVE = ?

[vENZi]

Hello virtualization fans !
We are trying to assemble datacenter with few servers in different location and yes we know the limitation for HA and etc. but we want to use it in other way.
We build datacenter with 2 servers in one location and we trying to add another one in remote location. We build Wireguard VPN between them all and after we add the third machine it see the main 2 machines and start exchanging the keys and data but at some point it stops and cannot complete the add to the cluster. The setup is like this :

[2 machines in one location] -> [router with Wireguard server] -> INTERNET <- [1 machine with Wireguard client on Proxmox itself ]

After research I realiase that Proxmox6 doesn't not use anymore multicast and it uses "UDP unicast" - I does not find much information about this protocol.
I am wandering is this unicast is the problem - can it pass trought Wireguard vpn or there is another problem ?
Please give me some ideas cuz I am stuck with this problem for 2 weeks already
Any help about remote machines datacenter will be helpful a lot !
Thanks and wish you productive day

 

[vENZi]

Anyone ?

 

[Alwin]

We are trying to assemble datacenter with few servers in different location and yes we know the limitation for HA and etc. but we want to use it in other way.

What does the 'use it in other way' mean?

After research I realiase that Proxmox6 doesn't not use anymore multicast and it uses "UDP unicast" - I does not find much information about this protocol.

This is the least of the problems. Varying latency and bandwidth will be more of a problem. Best use two separated clusters.

 

[vENZi]

What does the 'use it in other way' mean?


This is the least of the problems. Varying latency and bandwidth will be more of a problem. Best use two separated clusters.

We just want to use unified UserInterface between all of our servers and to be able to migrate machines between the servers easy when is needed.
Please can you tell me what are the requirements of this UDP unicast because we trying to do it over Wireguard tunnel but we have a difficulty to connect remote machine to the datacenter.
If layer 2 network is requerd for UDP unicast we will can build the vpn with OpenVPN.
I think as more is flexible Proxmox with such a remote setup as more people will find it useful for a lot of special cases.
Thanks

 

[Alwin]

We just want to use unified UserInterface between all of our servers and to be able to migrate machines between the servers easy when is needed.

That's why I added:

This is the least of the problems. Varying latency and bandwidth will be more of a problem. Best use two separated clusters.

If you don't have at least 1 GbE with local network latency between the two datacenters, clustering will not work as expected. Doesn't matter what VPN technology you use.

 

[BobhWasatch]

I am wandering is this unicast is the problem - can it pass trought Wireguard vpn or there is another problem ?

That just means it uses regular UDP, not multicasting as before. Same as video streams and a lot of other stuff. It will work fine over Wireguard but as @Alwin says there are other requirements beyond just getting packets through.

 

[vENZi]

That just means it uses regular UDP, not multicasting as before. Same as video streams and a lot of other stuff. It will work fine over Wireguard but as @Alwin says there are other requirements beyond just getting packets through.

Hmm I saw few posts about people that did that remote setup and all works despite HA , LM and shared storage ... here is example :

"I am running PVE cluster over WAN (different datacenters across the globe). It worked all the time flawlessly and best suited my needs (of course no shared storage, LM or HA but still central management, easy offline migrations etc). Some time ago I've upgraded to PVE 6.0 and was able to run the corosync directly through WAN unicast interfaces, no need to build VPN which is not necessary for some of my nodes. Simplified my setup and I was glad"

In this link : "https://forum.proxmox.com/threads/pve-6-0-corosync-over-wan-high-latency-looses-sync.59761/"

I want to make the same thing and if I see someone succeed I thinking it is possible with some limitations. Is that possible ?

 

[Alwin]

Hmm I saw few posts about people that did that remote setup and all works despite HA , LM and shared storage ... here is example :

The message reads differently. The user is not using shared storage, he is not using live migration and he is not using HA. The only thing he does is to use offline migration and central management.

He further states:

But now sporadically I have some kind of corosync "sync" problems. When there are some (even short time!) connection problems between nodes (which is understandable and unavoidable on WAN links) cluster seem to get broken. When I notice this I simply run:

And those are exactly some of the problems you will run into.

 

[e2a0fd36dc210fe]

Unless I'm mistaken, I feel as though he only wants to have central management and offline migration, just like the user who's post he is referencing. I don't understand why we're being so flat with the guy, lol.

At the risk of getting the same responses, myself... I found this thread while trying to determine if it's possible to initiate clustering using the wg0 link created based on my WireGuard config because I, like Mr. vENZi, want to have a cluster of nodes running in different geographic locations through a WireGuard tunnel for the sake of centralized management and potentially offline migrations, but nothing else. I do intend to install additional systems at the main office of my business on a 1G/10G switch so that those systems specifically can benefit from the full additional capabilities that come with clustering. Alas, it remains unclear as to why the wg0 interface isn't showing up in the 'Link' section of the cluster setup dialogue - does this section list physical adapters only? Is there any way, through config files or otherwise, to get the wg0 interface to show up here?

 

[tjcomserv]

Hello, have you tried adding the wg0 interface to a new bridge?