Question about cluster with nodes in different physical sites

S

svcoroe

New Member
Feb 25, 2025
2
0
1
Hello,

We are currently testing Proxmox and have two physical sites connected via the same network but with different gateways.

We already have a VMware cluster successfully running between these two sites.

For our Proxmox setup, we initially created:

  • Site A: 2 Proxmox nodes
  • Site B: 3 Proxmox nodes
At first, we set up two separate clusters—one for the nodes in Site A and another for the nodes in Site B—without issues.

Then, we decided to reinstall Proxmox on all nodes and attempt a single cluster spanning both sites. We created the cluster on one of the nodes, but as soon as we tried to join a node from the other site, everything stopped working. The web interface became inaccessible, leaving only SSH available. We had to dismantle the cluster before regaining control of the nodes.

We’ve tried a few times, and the results are always the same, only a cluster with the nodes in the same physical site works
  • Latency between sites: ~4-5 ms
  • Ports open: UDP 5405-5412, TCP 22
Before diving into troubleshooting, we’d like to confirm: Is this setup theoretically possible, and should it work as expected?

Thanks in advance for your guidance!
 
short answer: don't.

semi short answer: the only sane "multi site" recipe for systems involving data integrity is a 3 site solution. What are you trying to achieve?
 
Thanks!

We aimed to set up a cluster to manage all nodes from any logged-in server and to migrate VMs between them. we don’t have shared storage, each server has its own disks.
We can still set up a cluster at each site and migrate VMs between nodes within each cluster then.
 
svcoroe said:
Thanks!

We aimed to set up a cluster to manage all nodes from any logged-in server and to migrate VMs between them. we don’t have shared storage, each server has its own disks.
We can still set up a cluster at each site and migrate VMs between nodes within each cluster then.
Click to expand...
It's not usual to be disk jockeying VMs between two datacenters, with complex networking, things break (MPLS, IPSEC, etc).

For DR I've used ZFS replication successfully between sites.

For a "single pane of glass" solution to manage both sites, you're probably better served by the new Datacenter Manager (I haven't tried it yet):

https://pve.proxmox.com/wiki/Proxmox_Datacenter_Manager_Roadmap

t.lamprecht

Thread 'Proxmox Datacenter Manager - First Alpha Release'

We’re excited to announce the alpha preview of Proxmox Datacenter Manager! This is an early-stage version of our software, giving you a first impression at what we’ve been working on and a chance to collaborate.

What's Proxmox Datacenter Manager?
The Datacenter Manager project has been developed with the objective of providing a centralized overview of all your individual nodes and clusters. It also enables basic management like migrations of virtual guests without any cluster network requirements.

The project is fully developed in the Rust programming language, from the...
  • Like
 
Last edited:
You must log in or register to reply here.
Top Bottom