Multi-site cluster dns config

[hugolet]

Hi everyone!

This is my first post here and sorry in advance English is not my native language.

It's been two years now i'm using Proxmox at home. And it work like a charm!

But now i want to go on more complicated installation. And on some points I don't really know how to do this...
I have 3 proxmox servers in 3 different places across the city, connected into a cluster via Tailsacale. I had some hard time to set up the cluster but now it is working great!
I tried to creat a Ceph sharing beetwen them but as I read it, it probably won't work because of the latency. And this is the case... OSD go off/on in/out all the time... If ever someones succed into creating a Ceph Cluster between multisite proxmox, i could be interested.
For the moment i use ZFS sharing and it work's great. Replication and migration also work great.

But this is not the purpose of today!

From here i used the Nginx Proxy Manager for my home node with some duckdns and OVH Dyndns configured into my box and NPM.

I would like to know how you would do to expose some VM/CT to internet with dynamic IP on such a cluster?

would you use Pfsense? Nginx? Other? One on each machine? One on each CT/VM? Wich configuration on proxmox would you do (Host/DNS...).
The aim is to keep access to services with is web address regardless of the machine on which he is.

Sorry it's a bit of a blurry question... But I read so many things, but no one really responding to my needs, that I don't even know where to begin...

If you could guide me on where to start!

Thanks!

 

[aaron]

I tried to creat a Ceph sharing beetwen them but as I read it, it probably won't work because of the latency. And this is the case... OSD go off/on in/out all the time... If ever someones succed into creating a Ceph Cluster between multisite proxmox, i could be interested.
For the moment i use ZFS sharing and it work's great. Replication and migration also work great.

Ceph is very latency sensitive as you have multiple network trips for each IO operation. Even if you have your own dark fiber in between locations, latency can easily be too high for good performance. For a cluster that is a bit more spread out and has higher latency, local ZFS with VM replication is a much saner approach. It is async and so it does not affect guest disk performance.

Regarding your networking/DNS question: hmm. I personally would try to have one box somewhere that is the ingress/egress box. Ideally with a fixed IP. That could be a small VPS that is part of the overlay network (Tailscale).
Using dynamic IPs with DNS record updates could work as well, but depending on the services you want to run on them, a static public IP that is not associated with a range used by ISPs for private customers helps a lot. Especially if you consider sending out emails.