Support for clustering using wireguard

[jackintosh157]

I have multiple proxmox hypervisors on dedicated servers you can get from cloud providers. Each get their own public ip. I've linked my hypervisors up using wireguard so that have a private ip they can reach each other with.

The problem is, and I've read this somewhere else on the forums: that clustering only officially 100% works with nodes being on the same physical layer 2 network.

Adding a node to a cluster using a private wireguard ip instead of its public ip is possible. You have to manually edit the corosync.conf to use the private wireguard ip once the node tries to join the cluster. Unfortunately, this only solves some issues. I use a firewall to block all traffic coming into each node in the public ip address (accept ssh). It seems its impossible to change /etc/pve/.members, which always contains the public ip of each node (even though you can hack corosync.conf to work, .members is read only). Because I block all incoming ports on the public ip, some functions such as accessing info on another node on the webgui or SPICE terminal do not work, as they get blocked. Is there anyway to hack together something so that the nodes try to access the other nodes private ip instead of their public ip? Obviously it seems proxmox has yet to add official support for L3 clustering.

I believe one hack to fix this would be to dnat the destination public ip to the private wireguard ip of every other node, this would need to be configured manually on each node. Alternatively, just allow the public ip of every other node in the firewall, but this means their vms may have access to the ports of the hypervisor.

 

[jackintosh157]

Indeed using dnat works: heres a script you can setup that will add rules to dnat the public ip to each private ip

/usr/local/sbin/proxmox-wg-routing.sh:

#!/bin/bash

declare -A NODE_MAP=(
[pve01_public]=pve01_private
[pve02_public]=pve02_private
)

for PUB in "${!NODE_MAP[@]}"; do
WG="${NODE_MAP[$PUB]}"
echo "dnatting $PUB to $WG"
ip route replace "$PUB" via "$WG" dev wg0
iptables -t nat -C OUTPUT -d "$PUB" -j DNAT --to-destination "$WG" 2>/dev/null || \
iptables -t nat -A OUTPUT -d "$PUB" -j DNAT --to-destination "$WG"
done

Replace the pve0X_public and pve0X_private addresses. You could probably have just one script deployed to your pve nodes if you did a check for what the current public ip is (so you don't dnat your own public ip), but I'm just modifying the script for each individual node.

Then I added this systemd service /etc/systemd/system/proxmox-wg-routing.service:

[Unit]
Description=Proxmox custom WireGuard routing and DNAT
Requires=wg-quick@wg0.service
After=wg-quick@wg0.service
Before=pve-cluster.service pvedaemon.service pveproxy.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/proxmox-wg-routing.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

I also added to the AllowedIPs section of my wireguard the other public ips of the pve nodes. Note I am using a centralized wireguard server for these clusters, I don't need much bandwidth so this works. If you are directly peer to peer connecting the nodes over wireguard then you'll need to add some more logic for the dnat to allow traffic on the wireguard port to go to the actual real public ip. This setup will work for other L3 vpns as well.

 

[gurubert]

Keep in mind that clustering only works with latencies below 5 milliseconds.

 

[spirit]

The problem is, and I've read this somewhere else on the forums: that clustering only officially 100% works with nodes being on the same physical layer 2 network

since corosync 3.X, it's use unicast udp (before it was multicast), so it's working on layer3 network. (I'm using it in a full l3 datacenter).

so, it could work through internet public ip, but be carefull of latencies !