Shared Storage recommendations for 3-node HA Cluster

[gdi2k]

I am new to Proxmox, and in the coming weeks I will be setting up a Proxmox VE 4.x HA cluster with 3 servers. This means I need shared storage.

What shared storage configuration works best / is recommended by long-time users? I will not be using SAN or external NFS servers, only the RAID disks within the servers themselves. I would like to use something that is tried and tested, even if it slower or less efficient than some cutting edge solutions. The servers are E3 v3 Xeons, so 32 GB is my RAM limit per server.

I've been reading the wiki and a lot of forum posts (most of which want to do 2 node set ups, not 3). It seems to boil down to DRBD9 vs Ceph vs GlusterFS, with LVM on top or underneath (?) each.

I will only be using KVM images, if that helps.

Thanks!

 

[LnxBil]

I think, Ceph and GlusterFS are the only viable options for you situation. Ceph's hardware requirements are higher and the setup is a bit more complex, whereas GlusterFS is easier and you do have "real" files afterwards. Both do not include LVM on top (underneath maybe).

If you'd go with DRBD, then you have LVM on top.

In the end it comes down to how familiar you are with the techniques and how fast you can solve problems without destroying everything. It'd be best to try out all three and deliberately crash nodes to see how to handle failures. I'd never use a HA system without trying to break it in the first place. It's not a matter of if something happens, it's a matter of when (including human error).

 

[dietmar]

I would also recoomend ceph.

 

[gdi2k]

Thanks. I have played with GlusterFS previously with good success (unrelated to Proxmox), but never used Ceph. I will look into that in more detail on your recommendation.

 

[gdi2k]

I've been reading some more, and am seeing a lot of recommendations for using DRBD9 for my type of scenario. Is there some reason why I shouldn't consider it? Is it still unstable at this point - I saw that it is marked as a "technology preview" in the wiki for Proxmox VE 4.x series.

I also read this great wiki article:
https://pve.proxmox.com/wiki/DRBD9

 

[CBdVSdFSMB]

Is it still unstable at this point

Unfortunately yes. My last tries some months ago (April/May) ended in destroyed virtual disks. The guys at Linbit know about the issues, I have been in contact with them but the last bit of information I got was, that the problems had gotten more complicated cause they added more features to their agenda for the stable release - unfortunately there's no end in sight at the moment.

As you're planning to make a 3-node setup, Ceph is definitely a good alternative.

Cheers, Johannes