Geographically Redundant Proxmox Cluster

[henning]

Hi all,

I'm interested in building a geo redundant cluster that makes container migration/failover from one physical location to another possible. Working with two separated locations introduces limitations and I'm trying to figure out if a proxmox cluster can be deployed in this situation.

Where I'm coming from:
Currently I have a two node active/passive cluster in one rack that is running about 200 small OpenVZ containers. If the active node fails the passive node mounts the DRBD device and starts all containers. It's a self made cluster not using proxmox. It's running fine for about 5 years now.

Where I want to go:
By spreading my nodes across two physical locations I want to protect my applications from the outage of one location. Therefore I will have two racks each in a different data center. Both racks are connected with a dedicated 10 GBit/s fiber link over a distance of about 20 kilometers. Both racks are also connected to the public Internet with a 100 Mbit/s link (1 GBit/s possible). My public IPs will be routed to both locations so a node can allocate an IP no matter in which location the node is. Routing of IPs is none of my business and is done by a supplier. OpenVZ will be replaced by LXC.

This raises some questions:

- I think three nodes is the minimum to run a proxmox cluster. From a performance point of view, one active node is currently enough to run all my applications. Does it make sense to put two nodes in one location and the other node in the second location?

- Is it possible to do data replication with ceph with a 10 GBit/s fiber link over 20 kilometers?

- Both locations can only communicate over two paths with each other: a) The public Internet and b) the 10 GBit/s fiber link. Is that a show stopper because I will see unwanted fencing if one link dies?

It seems like the setup I have in mind is pretty unusual and I can't find a lot of information about the setup necessary to failover vms/containers between two separated location. Would be nice to get some thoughts from the proxmox experts.

Best,
Henning

 

[udo]

Hi all,

I'm interested in building a geo redundant cluster that makes container migration/failover from one physical location to another possible. Working with two separated locations introduces limitations and I'm trying to figure out if a proxmox cluster can be deployed in this situation.

Where I'm coming from:
Currently I have a two node active/passive cluster in one rack that is running about 200 small OpenVZ containers. If the active node fails the passive node mounts the DRBD device and starts all containers. It's a self made cluster not using proxmox. It's running fine for about 5 years now.

Where I want to go:
By spreading my nodes across two physical locations I want to protect my applications from the outage of one location. Therefore I will have two racks each in a different data center. Both racks are connected with a dedicated 10 GBit/s fiber link over a distance of about 20 kilometers. Both racks are also connected to the public Internet with a 100 Mbit/s link (1 GBit/s possible). My public IPs will be routed to both locations so a node can allocate an IP no matter in which location the node is. Routing of IPs is none of my business and is done by a supplier. OpenVZ will be replaced by LXC.

This raises some questions:

- I think three nodes is the minimum to run a proxmox cluster. From a performance point of view, one active node is currently enough to run all my applications. Does it make sense to put two nodes in one location and the other node in the second location?

Hi Henning,
for disaster recovery it's makes sense - but you will lost quorum and so you must manualy reconfigure the remaining node to start the VMs there.

- Is it possible to do data replication with ceph with a 10 GBit/s fiber link over 20 kilometers?

should be, but with ceph you have the next quorum-problem. I would suggest DRBD for this.

- Both locations can only communicate over two paths with each other: a) The public Internet and b) the 10 GBit/s fiber link. Is that a show stopper because I will see unwanted fencing if one link dies?

fencing? This mean HA?! HA with two DCs looks not easy for me...

It seems like the setup I have in mind is pretty unusual and I can't find a lot of information about the setup necessary to failover vms/containers between two separated location.

failover container mean you need the container on shared storage (nfs, or wait for pve4).
In case of nfs it's must accessible from both DCs... not easy if the connection breaks...

Udo

 

[henning]

I think you are right. Quorum will be a problem with two DCs. I will reconsider my setup.

Thanks for your input. Its very helpful to me!

 

[snpz]

Hi!
I have pretty similar case.
At the moment i have a 4 server proxmox cluster hosted in DC-A. As a shared storage i'm using CEPH.
And there comes DC-B, connected using fiber optic.
So my idea is to join to existing cluster another 4 servers (similar as located in DC-A) located in DC-B, expand my CEPH storage and configure fencing and HA using priorities. As result there is going to be 8 server cluster using shared CEPH storage.
One more thing - there is no option to get multicast between DC's, only unicast will work, but all nodes will be in the same network.
I guess this could be the biggest problem for cluster, right?
So my question is:
1) will this setup work as expected?
2) is there anything i forgot and that is vital?

Thanks a lot in advance!