Storage Model, Ceph vs GlusterFS

[mantisgroove]

Hello all,

'Quick question.

So, I know that Proxmox VE includes both Ceph, and GlusterFS support... however, I get the impression (and correct me if I am wrong on this) that Ceph is being pushed as the de-facto choice for HA/Clusters needing shared storage.

Red Hat however seems to favor GlusterFS for use cases like this, with Ceph being more suited for large Object Store systems in an OpenStack deployment.

Is there any particular reason for this?

I'm just curious, and would love to hear from anyone who has had positive, or negative experiences with either Ceph or GlusterFS in a shared storage / cluster setup...


...and while I have your ear, I'd also LOVE to hear from anyone who has experience or knowledge regarding my two absolute DREAM setups.

#1 - I have long used ZFS over NFS for storing both my actual virtual machine images/vmdk's, and their service storage (mail server mailstore/db). The ability to create volumes that stripe across multiple underlying mirrors (such as, say 5 separate vdev's which consist of 3 drives mirrored each), allows for much faster rebuild times when swapping out drives, much better overall volume performance than a traditional RAIDz2 / RAID6 volume, and the copy on write, full volume data-checksumming, and snapshotting make it as close to perfect for direct-attached storage that I can imagine. I LOVE it. However, I really, REALLY, long for an actual SAN system... such as say Quantum's StorNext cluster-filesystem (Apple uses it under the marketing name "Xsan" also)... that would allow for SAN like no single point of failure setups, BUT, ALSO have all of those ZFS style features. I haven't come across it yet.

#2 - I really, REALLY can't stop longing for the day when, macOS Sierra (or as time moves forward, whatever the current macOS is) can run on Proxmox in a fully supported, solid, and stable manner, not requiring special work-around's. Contrary to popular belief the EULA does NOT prohibit this. VMware's underlying hypervisor used by vSphere, ESXi, and Fusion, has supported this for nearly a decade now. You just need to be using Apple hardware underneath to satisfy the licensing.

So anyway... I'd love to hear what any of you think regarding Ceph vs GlusterFS, or also in regards to my two dream setups.

 

[czechsys]

Do you need block or file storage?
Gluster has one big disadvantage now - replication is done by client. So, even if you will have dedicated gluster cluster, you will use cpus on the hypervisors.

 

[Alessandro 123]

Do you need block or file storage?
Gluster has one big disadvantage now - replication is done by client. So, even if you will have dedicated gluster cluster, you will use cpus on the hypervisors.

not if you use export volumes via NFS.
anyway, replication is not cpu intensive, is much more limited to bandwidth.

You saturate the network much before than CPU

 

[mantisgroove]

I don't really care whether it's block or file storage providing it performs well. The "idea" is, I would really like to be able to have a cluster of say 3 hypervisors (Proxmox VE), and the virtual machine images (VMDK equivalent, raw, qcow or whatever) stored in some fashion where if one of the servers blows up, the vm data will be accessible to the other cluster members so I could re-launch it. From the perspective of the hypervisors, I'd like them to work exactly as they do in some of my other setups where they all just mount an NFS share from a 4th server (that performs great), but without having that single point of failure of the 4th server. If absolutely necessary I could even deal with having a group of three storage servers (1, 2, and 3 as hypervisors, and 4, 5, and 6 as a storage cluster), but I'd imagine there's really no reason I couldn't do that on the hypervisors. It's ok if replication took some cpu and bandwidth. I could just get more powerful hardware, or more cluster members as needed.

What it all comes down to is, I don't want to have to have specialized hardware (i.e. active/active Fibre Channel SANs), or proprietary commercial, or overly complex software. My conversations with Red Hat engineers so far have lead me to believe that Ceph is really ideal for Object storage, and GlusterFS ideal for, exactly what I'm trying to do.

I've had a lot of experience using a 4th server for storage that ran ZFS on FreeBSD and shared out via NFS (FreeNAS for instance makes that dead simple). From the perspective of the perspective of the hypervisor cluster, this setup works absolutely flawlessly. Performance is great, it's stable, the drive mirroring and striping, snapshotting, and full data checksumming... everything is PERFECT. Except for the fact that that 4th file server is a single point of failure. I can create a 5th and 6th even with some scripted automation to snapshot and replicate the data, but the promise of a single unified namespace or "volume" from the hypervisors perspective along with distributed redundancy is really just the "Holy Grail" I can't shake searching for. I've done a few quick tests of GlusterFS over the years and the performance was usually terrible (this was a while back though). I tried using it in Proxmox somewhere around v4.0 or so and the replication seemed inconsistent.

I'm sure there's no perfect answer, but I was just hoping to hear what some other people are doing, and what their results have been like so I don't have to re-ivent the wheel.

Thanks all! I'd love to hear anyone's experiences.

 

[Joao Correa]

Hello
I'm starting studies for adoption of Proxmox 5 with distributed Storages.
Very interesting this post and I share the same impressions.
Does anyone, including MantisGroove, use Ceph or Gluster and could you explain their experience?

 

[devinacosta]

I want to first start off by mentioning that I am a Red Hat Certified Architect, a rather short list of those in the world who have achieved this, and I mention it because being an RHCA I drink the "Red Hat" cool-aide if you will. Over the past year I have gone down a very similar path where I was looking at various Virtualization platforms and trying to decide what was the best storage solution for the various options that I tested.

I initially went down the path of using oVirt (Red Hat's upstream project for RHV) and was pushed towards the GlusterFS route was RHV does not support Ceph at this point in time. What I quickly learned from oVirt/RHV was that GlusterFS is a nightmare and is a beast when it comes to healing the filesystem. There will be many articles that will claim that you can do a 3-node Gluster Cluster however I highly advise against it and in fact I would say stay clear of Gluster. In the end I migrated off of oVirt and shut down Gluster as quickly as I could.

That then lead to me look at VMware which was to costly for my needs and Proxmox which I have heard quite a bit of buzz about. I can hands down tell you that Ceph is where it is at and what I can't recommend to you strongly enough. Ceph was created by a company called Ink Tank and then later sold to Red Hat, so Red Hat does now own Ceph and is currently maintaining the code. Ceph just plain and simple works when setup correctly, I have pulled out nodes left and right and brought them right back into the cluster and everything just worked beautifully. If i did the same thing in Gluster it literally took me one time an entire week working with Red Hat support to get my Gluster back into a healthy state. So in my opinion Ceph is the ONLY option to consider. It's very robust, resilient and has been amazing for me. I currently run 3 Proxmox clusters and they all run Ceph (either doing hyper-converged, and one doing fully dedicated ceph nodes).

With the new release of Luminous Ceph changed the architecture so that it now increased in speeds 2X from pervious versions of Ceph. So I highly recommend that you use Luminous when you deploy your Ceph infrastracture.

I don't know how many hours I lost while I was running Gluster where I literally was close to a major disaster, and I have never had any worries like that with Ceph.

Devin Acosta, RHCA

 

[Joao Correa]

Thanks for your report.
I saw that the Proxmox 5.1 version recently features the Ceph Luminous 12 LTS ready for production.

 

[DerDanilo]

I know this post is a bit old.

You might want to checkout Quobyte.
It's closed source and you need to pay for it but it's definitely worth it's money.
Compaired to other systems it's pricing is even pretty cheap and these guys know what they are doing.

 

[Vladislav Solovei]

So, as for now (2019), GlusterFS is still not recommended for small (3 x nodes x 6 bricks) cluster?

 

[Mike Lupe]

I would simply like to say I went down almost exactly the same path as @devinacosta (unfortunately I'm no RHCA . Healing Gluster on oVirt/HCI took me on several occasions either a week+ to get in a healed state, or even redeploying the entire HCI cluster. Since about an year, after having moved to Proxmox with HCI/Ceph life seems bright again.

One of those issues I was looking forward to with a Ceph HCI is you can "simply" add an additional storage node and total storage rises. On Gluster you _always_ have to add a number of nodes in multiple of your replica. E.g. when having a 3-node gluster cluster with replica 2 or 3 you need to add 3 additional nodes to crank up storage.

 

[davidlee]

I would simply like to say I went down almost exactly the same path as @devinacosta (unfortunately I'm no RHCA . Healing Gluster on oVirt/HCI took me on several occasions either a week+ to get in a healed state, or even redeploying the entire HCI cluster. Since about an year, after having moved to Proxmox with HCI/Ceph life seems bright again.

One of those issues I was looking forward to with a Ceph HCI is you can "simply" add an additional storage node and total storage rises. On Gluster you _always_ have to add a number of nodes in multiple of your replica. E.g. when having a 3-node gluster cluster with replica 2 or 3 you need to add 3 additional nodes to crank up storage.

great！！you guys help me so much,I'm looking the storage solutions for proxmox HCI now, and I can't decide myself ,after I saw the series discussion， now I knew the correct answer ! thank you so much !!