Proxmox cluster through NAT

[rcd]

I have a Proxmox server set up in my office, using private address range, for testing and learning.
Next step is to add another Proxmox server in the wild, i.e. on the big bad internet.
Now, my thought is I'd like to be able to create a VE on my test setup and once I'm happy with it, move it to the "production server" on the internet. I've already testet this inside my private network, now the question is, can I make this work through the NAT? I'd expect it would be a matter of forwarding a couple of ports in my pfsense router, is that correct?

 

[Lope]

I tried this with:
Public node: (public IP)
Private Node: NAT LAN IP (SSH port, and port 8006 forwarded to it)

When I try create the cluster and let Proxmox figure it out, it can't.
The public node tries to ping the LAN IP, which obviously fails.

I edited the Private Node's IP to be it's router's public IP.
But then Proxmox shows this error in the log
corosync[4962]: [KNET ] heartbeat: Unable to send ping (sock: 30) packet (sendto): 22 Invalid argument. recorded src ip: src port: 5405 dst ip: dst port: 5405

Because Proxmox is trying to ping from the interface that has that IP, but none of it's interfaces have the public IP.

I came up with a workaround, which works for short periods of time only.
In a nutshell it involves incrementing the corosync.conf files to the SAME new version number, then leave the LAN IP in the private node's conf, but put the private node's router's public IP in the public node's conf.

Please note, everywhere where I've said "IP" concerning the private node, is actually my router's dyndns hostname, not an IP, because it's a dynamic IP.

Steps to reproduce: (do everything below on both nodes)
1. systemctl stop pve-ha-crm; systemctl restart pve-ha-lrm; systemctl stop corosync pve-cluster; systemctl stop pve-cluster
2. edit /etc/corosync/corosync.conf (on both)
3. On the computer with the public IP, set the ring0_addr to the public IP (of the router) of your remote node.
3. On your node that has a private LAN IP behind a NAT router, set it's own ring0_addr to the LAN IP.
4. set config_version:to a higher number than it was, make it the same for both, so that they don't overwrite each other.
5. systemctl restart pve-ha-crm; systemctl restart pve-ha-lrm; systemctl restart corosync pve-cluster; systemctl restart pve-cluster

When I restart everything as above, it works. The nodes sync to each other. I can control the one Proxmox by connecting to the web interface of the other, etc. Then I go away, come back hours/days later, they're disconnected. Restart everything, they connect.
So my above solution is not working completely yet.

I've got more urgent things to work on so I'm not investigating further right now.
But I would love to see Proxmox staff comment on the above.

EDIT: Oops! I noticed I had posted my comments about this in an inappropriate thread.
I've moved them here now.

 

[rcd]

Well, obviously don't want to waste a day testing this if this is something that "everybody knows" is not possible, or for whatever reason isn't feasible. Seems from the above that at least I'm not the only one who think it's worth a try.

https://pve.proxmox.com/wiki/Firewall lists the following ports used by Proxmox:

Web interface: 8006
VNC Web console: 5900-5999
SPICE proxy: 3128
sshd (used for cluster actions): 22
rpcbind: 111
corosync multicast (if you run a cluster): 5404, 5405 UDP

I'm only interested in the clustering functionality, so I supposed the only NAT gateways needed should be for corosync? Do the Proxmos nodes communicate with each other through the web interface? (API perhaps?) .

Given I already have a lot of servers set up in the NAT, and I never use port 22 for ssh, is it possible to configure Proxmox to be aware of this (i.e. connect to a different port?)

 

[Lope]

rcd: Thanks for that list of ports. That's helpful. I literally only allowed port 8006 through. That explains why my nodes would sync for a while, then after a while show that they were disconnected. So basically what I did should work, just need to forward and allow the rest of the ports.