Best Proxmox Scalable High Availability Setup and Configuration

[hisaltesse]

I am trying to setup the best minimum scalable HA setup and Configuration. I read the documentation on High Availability, however I have some questions that are not well discussed in the documentation.

I was thinking of starting with 4 servers:
2 x front end server that will use a central storage (so I can do some load balancing and HA failover)
1 x central storage server for VMs
1 x backup server/storage. (to both replicate the central storage and also do daily/weekly/monthly backups)

It is my understanding that I need at least 3 nodes for High Availability. My questions are:

Q1: Does the third node (central storage) have to run promox or can it be just a NAS (running FreeNAS)?

Q2: For backups, is it better to have a dedicated NAS (with FreeNAS) or just another Proxmox server with a storage pool dedicated to backups?

Q3: What is the most efficient way to incrementally replicate the central storage to the backup server/storage?

• Is it by just creating incrementally a replica schedule in Proxmox UI or else?
• Is it better to replicate VMs to the backup storage (per VM/CT replication) or to replicate the entire central storage pool to the backup storage (in case the central storage goes down)?
• Can Proxmox replicate an entire storage pool or just VMs/LXCs?
• Is the Proxmox UI replication at the block level? And is it filesystem dependent (ZFS vs Ext4)?

Q4: How can I failover to the backup of the central storage if the central storage server goes down?
I understand Proxmox can do automatic VM failover if one of the nodes running the VMs/LXCs go down and the VMs/LXCs would auto start on the other node. However what if the entire central storage fails or goes offline, can Proxmox also failover to the replica of the central storage that is in the backup server/storage and have all nodes resume their VMs from there?

Q5: Or is the shared storage automatically replicated in other nodes in the cluster?

Q6: Is there a diagram that illustrate how to setup a 3-node High Availability Cluster with Proxmox?

Q7: Is fencing automatically setup for the system or do we have to manually set it up at different levels (switch, host etc.)

Please let me know and thank you for your input.

 

[alexskysilk]

Q1: Does the third node (central storage) have to run promox or can it be just a NAS (running FreeNAS)?

Yes depending on what services you want the node to provide it can be anything. If you want it manageable through the proxmox interface you'd want it running proxmox.

Q2: For backups, is it better to have a dedicated NAS (with FreeNAS) or just another Proxmox server with a storage pool dedicated to backups?

Same as above, with the caveat that you'd want the backup to be physically separate from the rest of the cluster- not much point if an environmental fault takes it out with your production gear...

Q3: What is the most efficient way to incrementally replicate the central storage to the backup server/storage?

zfs send/receive, but rsync can work if you dont have the option.

Q4: How can I failover to the backup of the central storage if the central storage server goes down?

if your datasets are zfs, proxmox features zfs asynchronous replication (https://pve.proxmox.com/wiki/Storage_Replication) having said that, have you considered ceph?

Q5: Or is the shared storage automatically replicated in other nodes in the cluster?

https://pve.proxmox.com/wiki/Ceph_Server

Q6: Is there a diagram that illustrate how to setup a 3-node High Availability Cluster with Proxmox?

https://pve.proxmox.com/wiki/High_Availability, but generally speaking its just three nodes interconnected with dedicated cluster interfaces.

Q7: Is fencing automatically setup for the system or do we have to manually set it up at different levels (switch, host etc.)

https://pve.proxmox.com/wiki/Fencing

In general, the forum is a great place to ask questions but you can find so much in the documentation...

 

[LnxBil]

The HA PVE provides (with at least 3 nodes) is best served with either dedicated shared HA storage (dual-head-multi-UPS SAN with multiple paths to your storage) or distributed shared storage provided by CEPH. Both provide no SPOF and provide fault tolerance and automatic failover.

 

[hisaltesse]

Q8: So how would ceph potentially work in Proxmox HA Cluster?
Would it be to setup 3 Proxmox servers, and combining each server disk/RAID into a ceph storage in such a way that the cephFS would show up as a single storage from each Proxmox server interface but will be using disks from all 3 nodes?

Q8.1 If this is the case then this would essentially replace the NAS right?

Q8.2 but if this replaces the NAS, wouldn't ceph running off of all 3 nodes create major overhead on the servers and network bandwidth? Any known benchmark showing how much additional memory, cpu and network bandwidth Ceph takes to keep all storage in sync?

Q9: Isn't Ceph storage better for just backups/ISO/templates and not for running live VMs?

 

[alexskysilk]

Q8: So how would ceph potentially work in Proxmox HA Cluster?
Would it be to setup 3 Proxmox servers, and combining each server disk/RAID into a ceph storage in such a way that the cephFS would show up as a single storage from each Proxmox server interface but will be using disks from all 3 nodes?

Just so, but you wouldn't be using cephfs- you'd be using ceph rbd which is ideal (and designed for) virtualized environments. cephfs can be ran concurrently and may be used for userspace files as you suggested in your followup question.

 

[hisaltesse]

Just so, but you wouldn't be using cephfs- you'd be using ceph rbd which is ideal (and designed for) virtualized environments. cephfs can be ran concurrently and may be used for userspace files as you suggested in your followup question.

Can both ceph and cephfs be setup and managed using proxmox GUI or just cephfs?

 

[alexskysilk]

both. have a look here: https://pve.proxmox.com/pve-docs/chapter-pveceph.html

 

[LnxBil]

Q8.1 If this is the case then this would essentially replace the NAS right?

Sure, this is your single-point-of-failure (SPOF) and you cannot have SPOFs in a HA environment.

 

[sektor]

Sorry to bring this topic back up to the top but I have some questions of my own as this is something I have been doing some serious research in.

My scenario is I have 3 servers that will be at a hosting center originally I was thinking about going with 2 freenas type servers and 3 proxmox hosts, but from what I am reading here a NAS for user data can essentially be accomplished with ceph fs and then regular ceph for VM replication?

The 3 servers will have 64GB of ram each but I can go higher 2 240GB ssd's I was thinking about putting in a raid 1 for redundancy and 4 1TB HDD's all will have a 10gb nic connected to a 10gb switch, also is there a pretty decent how to guide?

I want to use the 4 1TB HD's for data, iso, and vm storage that will be redundant across all 3 servers, perhaps expanding to more hosts.

 

[LnxBil]

1) Please open a new thread
2) don't private message people in this thread to signal them that you wrote something, the system will pick this up by itself
3) Have you read https://pve.proxmox.com/pve-docs/chapter-pveceph.html?

 

[sektor]

My apologies, I opened a new thread and have read through the documentation and in the process of rereading it again.