Hi!
I would like to set up a cluster with 3 nodes. 2 Local (same network), and 1 remote. The remote one is not that far, 5Km. With latency around 3-4ms. The local nodes have 0.5ms latency. All clusters have 10GBit both local and WAN connection (more like 7Gbit in real life usage).
I'm trying to cluster the remote node to the 2 local ones but I haven't got much success. I though it might be the latency as per old threads I found that it needed to be 2ms or less. But according to the lastest wiki, it works with 5ms or less https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network_requirements
What I've tried:
Node 1 (local) Network 192.168.10.0/24
Node 2 (local, this would be just a raspberry pi with proxmox installed for quorum)
Node 3 (remote) Network 192.168.1.0/24
Node 1: Set up a LXC Ubuntu 22.04 container, install pivpn with wireguard as a server (I followed this tutorial, but used 22.04), as it seems like it's the easiest way to setup wireguard. Generate a new "client" .conf file. Lest call it wg0.conf. Port forward etc. The vpn works as I've tried to add it to my phone and I can connect to the proxmox GUI just fine using LTE.
Node 3: Install wireguard in the proxmox host (as root):
apt install wireguard
apt install resolvconf
cd /etc/wireguard
nano wg0.conf (copy the generated data in the server)
*For some reason installing wireguard and resolvconf removes the DNS configuration from proxmox. So I have to add the domain and dns server again. I just used 1.1.1.1 or 9.9.9.9.
systemctl start wg-quick@wg0 (to connect)
**I think I have to install the client in the proxmox host as it need to not have any VM or Container in order to join as cluster.
This works, I can now ping from Node 3 to Node 1.
But when joining the cluster, I get this:
And this pop-up:
If I refresh now I can't login as it says wrong password (it's not)
The node appears on the cluster, but with a red sign:
And now when restarting the node, the proxmox install is borked. No GUI. I have to manually enter in the CLI this to recover the proxmox GUI:
Then I can enter the GUI again. But also the "local" and "local-zfs" have merged and only have "local" now. But the options of the "local-zfs" have merge into local (this is really weird to be honest).
What could be wrong? Is this a latency problem? I can add shared storage into Node 3 from the Node1 LAN with no problem.
Even if the performance is not really good I would like to try it.
Thanks a lot!
This are my first steps with Proxmox and my idea is to be able to have a redundant VM and/or Containers so if my node 1 fails, node 3 can take over.
I would like to set up a cluster with 3 nodes. 2 Local (same network), and 1 remote. The remote one is not that far, 5Km. With latency around 3-4ms. The local nodes have 0.5ms latency. All clusters have 10GBit both local and WAN connection (more like 7Gbit in real life usage).
I'm trying to cluster the remote node to the 2 local ones but I haven't got much success. I though it might be the latency as per old threads I found that it needed to be 2ms or less. But according to the lastest wiki, it works with 5ms or less https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network_requirements
What I've tried:
Node 1 (local) Network 192.168.10.0/24
Node 2 (local, this would be just a raspberry pi with proxmox installed for quorum)
Node 3 (remote) Network 192.168.1.0/24
Node 1: Set up a LXC Ubuntu 22.04 container, install pivpn with wireguard as a server (I followed this tutorial, but used 22.04), as it seems like it's the easiest way to setup wireguard. Generate a new "client" .conf file. Lest call it wg0.conf. Port forward etc. The vpn works as I've tried to add it to my phone and I can connect to the proxmox GUI just fine using LTE.
Node 3: Install wireguard in the proxmox host (as root):
apt install wireguard
apt install resolvconf
cd /etc/wireguard
nano wg0.conf (copy the generated data in the server)
*For some reason installing wireguard and resolvconf removes the DNS configuration from proxmox. So I have to add the domain and dns server again. I just used 1.1.1.1 or 9.9.9.9.
systemctl start wg-quick@wg0 (to connect)
**I think I have to install the client in the proxmox host as it need to not have any VM or Container in order to join as cluster.
This works, I can now ping from Node 3 to Node 1.
But when joining the cluster, I get this:
Code:
Establishing API connection with host '192.168.1.210'
Login succeeded.
check cluster join API version
No cluster network links passed explicitly, fallback to local node IP '192.168.10.2'
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service
And this pop-up:
If I refresh now I can't login as it says wrong password (it's not)
The node appears on the cluster, but with a red sign:
And now when restarting the node, the proxmox install is borked. No GUI. I have to manually enter in the CLI this to recover the proxmox GUI:
Code:
systemctl stop pve-cluster
systemctl stop corosync
pmxcfs -l
rm /etc/pve/corosync.conf
rm -r /etc/corosync/*
killall pmxcfs
systemctl start pve-cluster
Then I can enter the GUI again. But also the "local" and "local-zfs" have merged and only have "local" now. But the options of the "local-zfs" have merge into local (this is really weird to be honest).
What could be wrong? Is this a latency problem? I can add shared storage into Node 3 from the Node1 LAN with no problem.
Even if the performance is not really good I would like to try it.
Thanks a lot!
This are my first steps with Proxmox and my idea is to be able to have a redundant VM and/or Containers so if my node 1 fails, node 3 can take over.
Last edited: