Glusterfs is still maintained. Please don't drop support!

[kayson]

The PVE 9.0 Beta Release Notes states: As GlusterFS is no longer maintained upstream, Proxmox VE 9 drops support for GlusterFS storages.

But this is simply not true. Glusterfs is actively maintained. The last commit and release (11.2) happened just three weeks ago! Furthermore, Debian Trixie has recent packages for gluster server, client, and cli.

While I understand that glusterfs may be in an awkward spot for enterprise since Red Hat EoL-ed Red Hat Gluster Storage, that happened in 2024, and it has not stopped development of the open source project.

It is well known that ceph does not perform well with consumer hardware, so glusterfs fills a much needed use case for homelabs and small scale users.

Please don't drop support for glusterfs!

 

[ProKn1fe]

Their github not even close to be "active maintained". Some bugfixes and nothing more.

 

[kayson]

Year to date commit history

I counted 26, so an average of about 1 a week.

Releases are slower, but still pretty consistent: https://github.com/gluster/glusterfs/tags

 

[jlauro]

If it's not broke, even as low as a few commits a year can be considered actively maintained. It might not be active development, but that's not the same as not being maintained.

That said, I wouldn't think glusterfs to be good for vms, except maybe a few low I/O ones... what kind of performance (ie: fio benchmark) can you get inside of a vm? Maybe I am under estimating it for random I/O....

 

[kayson]

If it's not broke, even as low as a few commits a year can be considered actively maintained. It might not be active development, but that's not the same as not being maintained.

That said, I wouldn't think glusterfs to be good for vms, except maybe a few low I/O ones... what kind of performance (ie: fio benchmark) can you get inside of a vm? Maybe I am under estimating it for random I/O....

I did a bunch of profiling on my cluster when I set it up. I uploaded it here: https://pastebin.com/qpqa9LYr

For every test, I did it both with and without direct I/O. I tested raw drive performance, then compared GlusterFS to CephFS, testing on the Proxmox host. I did do one test using virtiofsd to pass the GlusterFS mounts into a VM, and I was going to do that with CephFS too, but the performance on the host was so bad I didn't bother. Let me know if any of the results aren't clear.

To summarize, CephFS is unusably slow. GlusterFS gives a pretty big performance hit on the SSD too, but it wasn't as bad. Because of the way the test script I used is set up, GlusterFS gets to take advantage of files getting cached in memory on other peers, even though I'm using direct/invalidating the cache. (This is why you see some results that are faster than the HDD's raw speed). Ultimately, CephFS was so bad, though, it's just not usable for me.

You do pay a price for GlusterFS's heavy memory caching, vs CephFS's constant fsyncing: it's riskier in terms of data loss. At least theoretically... I have everything on a UPS, so I'm ok with the tradeoff.

 

[jlauro]

I did a bunch of profiling on my cluster when I set it up. I uploaded it here: https://pastebin.com/qpqa9LYr

It appears (based on the path, but maybe I glanced too quick) that was tested based on the performance of the host. I would be more interested in seeing what the performance would look like to a vm running in proxmox... that could give significantly different characteristics of qcow2 vs block layers.

Or was that script run from inside a VM on proxmox? It's not completely clear.

 

[kayson]

It appears (based on the path, but maybe I glanced too quick) that was tested based on the performance of the host. I would be more interested in seeing what the performance would look like to a vm running in proxmox... that could give significantly different characteristics of qcow2 vs block layers.

Or was that script run from inside a VM on proxmox? It's not completely clear.

There are a bunch of different tests there. Most of them were on the Proxmox host. A few (which are noted as such) were run in a VM with virtiofsd mapping the directory into the VM.

Checking the host is best case scenario, though, right? Aside from caching reads in memory, how could performance in the VM be better than on the host?

 

[jlauro]

I consider checking the VM is closer to real world. I don't care what the host sees, I want to know what it translates to the vms. I don't expect it to be better then the host, but more importantly I also don't expect the VM to scale in performance with the host when comparing completely different filesystems like block vs file. Who cares if the host can do a million operations a second if from within the VM it can only do 10. Who knows, the way caches interact, it's possible that you could get better performance inside a VM. You could say I want to know worse case scenario, not best...

 

[kayson]

I consider checking the VM is closer to real world. I don't care what the host sees, I want to know what it translates to the vms. I don't expect it to be better then the host, but more importantly I also don't expect the VM to scale in performance with the host when comparing completely different filesystems like block vs file. Who cares if the host can do a million operations a second if from within the VM it can only do 10. Who knows, the way caches interact, it's possible that you could get better performance inside a VM. You could say I want to know worse case scenario, not best...

Fair enough. I agree with you in principle, though I do think that in this particular case, especially with virtiofsd, there's no way to make up the performance gap on cephfs. Nonetheless, I'm willing to try the benchmark. I just need to tear down my running glusterfs volumes, so it'll be a little project.

 

[alexskysilk]

Releases are slower, but still pretty consistent: https://github.com/gluster/glusterfs/tags

These results are too slow to be of any production value. on the basis of your benchmarks it wouldnt be worth pursuing.

It is well known that ceph does not perform well with consumer hardware, so glusterfs fills a much needed use case for homelabs and small scale users.

It really doesnt. Gluster doesnt have much value under a certain critical hardware deployment, and its only really suitable for file store- which at the "homelab" use case is much more effectively served with a NAS.

 

[kayson]

These results are too slow to be of any production value. on the basis of your benchmarks it wouldnt be worth pursuing.

You mean the benchmark results are too slow to be valuable in production? That's somewhat irrelevant to a homelab.

It really doesnt. Gluster doesnt have much value under a certain critical hardware deployment, and its only really suitable for file store- which at the "homelab" use case is much more effectively served with a NAS.

Your first point may be true. I don't really have much visibility beyond my own use case and what I see online. I disagree with the second, though. With a proxmox cluster, having a separate NAS is not only inefficient from a power standpoint, it also introduces a single point of failure, which I'm trying to avoid. It may be less common than a simple NAS and nfs, smb/cifs, etc, but I know I'm not the only one looking at hyperconverged homelabs - I got the idea from reading about others who've done it.

 

[fabian]

https://qemu-project.gitlab.io/qemu/about/deprecated.html#gluster-backend-since-9-2

Qemu will remove support (possibly during the 10.x release cycle)

 

[Johannes S]

It really doesnt. Gluster doesnt have much value under a certain critical hardware deployment, and its only really suitable for file store- which at the "homelab" use case is much more effectively served with a NAS.

Same is true for small businesses. Together with storage replication this should cover all usecases where the available ressources don't merit Ceph.

And it's also propably not worth it from a business point of view. Don't get me wrong: Homelabs are great, I'm a homelabber myself (at work we are stuck with VMWare unfortunely and I'm not even part of the virtualization team), but I doubt that they contribute to a significant amount of the Proxmox Server Solutions GmbH revenues. Their main income propably comes from companys who are looking for a cheapter alternative to VMware, Nutanix and co. For these customers stuff like the new datacenter manager, the Snapshot on LVM functionality (from the 9 beta) or other missing features are way more important than trying to maintain a fork of qemu for glusterfs-support.

Now I don't work for them and don't know their account books, so I might be wrong

 

[guruevi]

Theoretically, under the same conditions, Gluster and Ceph shouldn’t be that much different in performance. If your hardware and network is poor, so will performance of any networked hardware. Gluster pretends to be fast by read caching, but as you said, multi-client locks and invalidating cache quickly destroys that in real world scenarios., you can configure Ceph to have big read caches too. You also have to be careful as Gluster by default is set up to not have any redundancy so that is a big problem, as any node takes the cluster down, in comparison to Ceph that requires by default 3 copies to be distributed.

 

[t.lamprecht]

Yeah, QEMU dropping support was actually what put GlusterFS in the spotlight for things to reconsider for this release.
Maintaining downstream QEMU support is a huge amount of extra work that would need good justification, but we do not see the usage numbers for GlusterFS in our enterprise support evaluations that would justify that effort.

Showing some lightweight development activity after years of slowing down to a crawl is better than nothing, but evaluating the commits it's rather still a bit far away from what we need for enterprise support, that's why the decision to drop built-in support for Proxmox VE 9 will be upheld.

But note that Proxmox VE 8 will be supported for about a year after the final PVE 9.0 release, and GlusterFS will keep working there.
Additionally, you can still mount a GlusterFS storage manually and add it as directory storage to Proxmox VE 9.

 

[jlauro]

https://qemu-project.gitlab.io/qemu/about/deprecated.html#gluster-backend-since-9-2

Qemu will remove support (possibly during the 10.x release cycle)

I only seen reference to one little announcement, that they might possibly drop support, not that they will or have dropped support. Anyways, it does sound like those interested in glusterfs for proxmox should lobby qemu to remove it from it's list of depreciated features. I assume that would help proxmox to continue to support it if qemu continued to.

 

[petruzzo]

On the page of the qemu project dont tell about the release 10 but "in a future release" and they write "Unless the development gains momentum again"
maybe the support can continue after the new release

 

[fabian]

once something is deprecated in QEMU, it is only guaranteed to be there for one more release (see the note up top in the link I provided).

 

[kayson]

Yeah, QEMU dropping support was actually what put GlusterFS in the spotlight for things to reconsider for this release.
Maintaining downstream QEMU support is a huge amount of extra work that would need good justification, but we do not see the usage numbers for GlusterFS in our enterprise support evaluations that would justify that effort.

Showing some lightweight development activity after years of slowing down to a crawl is better than nothing, but evaluating the commits it's rather still a bit far away from what we need for enterprise support, that's why the decision to drop built-in support for Proxmox VE 9 will be upheld.

But note that Proxmox VE 8 will be supported for about a year after the final PVE 9.0 release, and GlusterFS will keep working there.
Additionally, you can still mount a GlusterFS storage manually and add it as directory storage to Proxmox VE 9.

Thanks for at least taking a look and considering it.

 

[crabbypup]

I have not used glusterfs "recently" (within the last few years), but I have used it in the past, and the results were unstable at best, abysmal at worst, and the management was miserable. We had consistent crashes of the daemons under benchmark loads, but even for backup storage it was finnicky and we found it unreliable and fragile.

The gluster mailing list is utterly dead, the website is dead, and the repo is more or less dead. it's in "critical maintenance fix" mode, which redhat said would happen.

I don't think ceph is unusably slow, but one critical issue that I see repeatedly is that people always set it up with very few OSDs, which is not what it was designed for.

In my homelab (or more accurately, home prod), I get very usable performance out of 24x 500gb 2.5" laptop SATA drives, paired with 8x 500gb SATA SSDs for DB/WAL across 4 hosts - which is not an especially unreasonable number of disks imo, and someone running fewer or less demanding workloads would likely be fine with half as many disks.

The RBD volumes on this ceph cluster back an elasticsearch + graylog cluster, with the bulk of the message volume being netflow from a pair of opnsense router VMs, so the workload is about 6-10x more write than read most of the time. I'm not really using cephfs as RBD volumes are more performant to begin with for backing VMs and containers.