Glusterfs is still maintained. Please don't drop support!

This has nothing to do with the question whether gluster is dead or not.
Yes it does. It's a new version.

But ( I might be wrong though ) my impression is that the whining on the loss of gluster mainly comes from homelabbers so for them the directory option might still be "good enough"
A rude way to make that observation, but I dont disagree. Nonetheless, directory is far less convenient that having the integration.
 
Yes it does. It's a new version.
A new version or "another attempt at keeping life support on"? This doesn't matter for qemus gluster support anyhow. Debian packaging new gluster versions wom't help in maintaining the Code for qemus support of gluster as storage backend.

A rude way to make that observation, but I dont disagree. Nonetheless, directory is far less convenient that having the integration.
You are missing the point: For continuing to support glusterfs Proxmox developers would have to maintain their own fork of qemu at some point. Since this will cost developer times ( which could be used for other things ) this makes only sense businesswise if there is demand from paying customers. Homelabbers usually don't pay developers salaries and can still use glusterfs via the directory workaround. I might be wrong but my impression is that the developers prefer to concentrate on features people miss from vmware/esxi
 
Last edited:
A new version or "another attempt at keeping life support on"? This doesn't matter for qemus gluster support anyhow. Debian packaging new gluster versions wom't help in maintaining the Code for qemus support of gluster as storage backend.


You are missing the point: For continuing to support glusterfs Proxmox developers would have to maintain their own fork of qemu at some point. Since this will cost developer times ( which could be used for other things ) this makes only sense businesswise if there is demand from paying customers. Homelabbers usually don't pay developers salaries and can still use glusterfs via the directory workaround. I might be wrong but my impression is that the developers prefer to concentrate on features people miss from vmware/esxi
You are the one missing the point. Qemu dropped the glusterfs storage backend claiming it's not maintained, which just isn't true. Sure, you could make an argument that it lacks sufficient support for enterprise deployments or even that it's poorly or slowly maintained. But you can't call something unmaintained if people are meeting weekly to discuss it.

Regardless, qemu not having a glusterfs storage backend is orthogonal to Proxmox VE supporting glusterfs as a storage plugin. They're completely independent things. Having the storage plugin means its usable as PVE "storage" in the GUI or CLI (the former requiring the interface elements to be patched back in). While this includes qemu VM images, it also includes isos, CT images, backups, etc. QEMU is just a small part. PVE also handles the mounts for you. There's a lot of convenience to gain over "directory." And really, a qemu storage backend for a file-level storage system doesn't amount to much anyways.
 
Not sure what the purpose of that statement serves on the proxmox forums. Probably should direct it to the QEMU devs.
Proxmox says they removed the storage plugin because it's dead because QEMU dropped the backend support (which QEMU did because they also say its dead). See post above for why the QEMU backend is only marginally important.
 
right; so logically the body responsible for the decision is elsewhere (QEMU.)

There's a lot of convenience to gain over "directory." And really, a qemu storage backend for a file-level storage system doesn't amount to much anyways.
directory support still works and requires no support from PVE anyway. If the QEMU storage backend serves no additional utility, then what is the actual request for?
 
Last edited:
  • Like
Reactions: Johannes S
right; so logically the body responsible for the decision is elsewhere (QEMU.)


directory support still works and requires no support from PVE anyway. If the QEMU storage backend serves no additional utility, then what is the actual request for?
no... proxmox is the responsible body for removing the storage plugin, which is (essentially) what this whole thread is about....

see post above, like i already mentioned
 
Most of the data you ask about is really irrelevant.

I thought I was reading a 10yo post. How do you get Win11 to run on a 4th gen Intel?

If you want a technical discussion, you need to do it on technical terms.

Your benchmarks (like most benchmarks) are invalid because you're not benchmarking the 'same thing'. Moreover, you're benchmarking 1 thing (Windows?) which, if you know anything about storage - Windows NTFS always tells the subsystem it is doing an async write (yeah, NTFS is bad for your data), whereas with Ceph + QEMU any write is sync.

You can find some benchmarks from 2013 when Ceph was really young that already showed Ceph already winning various major points against Gluster, IOPS, latency but not throughput necessarily. And Ceph has improved remarkably in 13 years while Gluster kind of stagnated, by 2020 Ceph was a consistent winner in most benchmarks. But those are synthetic benchmarks, often in real-world situations, not a homelab. On ancient hardware, there is all kinds of reasons modern code won't do well. On consumer hardware even worse. Consumer hardware has a tendency to lie about data consistency as well and (especially SSD) will cache writes 'by default', Ceph (like ZFS) tends to avoid those problems even on consumer hardware.

You see the opposite on real server hardware though.

You’re running Windows 11 on a CPU from the Windows 7 era. That proves absolutely nothing other than that you can get hardware to do insane things. I can show you a Ceph cluster that does Windows 11 updates in less than 5 minutes on just 10G backbone with SAS drives.

You own comments disagrees with your statement that the data I ask about is irrelevant.

(It is highly relevant, pursuant to your own statements. If they were irrelevant, then you wouldn't have written what you wrote.)

The point was to drive the system to some kind of unified performance peak, what CPU and whether virtual or hardware and exact test design is rather irrelevant
Says the person who states:

Comparing GlusterFS and Ceph on 15yo hardware is like running a VW Beetle, then transferring that engine into a modern car and comparing how fast they go. Sure, the Beetle will beat the new car in a speed race, but it will kill its occupants in a crash and be overall much less safe on the road today, and the Beetle still can't compare to the performance of your average modern car.

According to you, if it wasn't relevant, then you wouldn't have written/mentioned it. But you did. Therefore; even according to you, it is relevant, otherwise you wouldn't have mentioned it in the first place.

Windows 11 isn't even supported on non-SSD systems, and there is a reason for that.

If it isn't relevant, then why would you even bother to mention any of the points that you've mentioned?

If your own points weren't relevant, then why mention it in the first place?
so you can make a fair comparison of performance without any tricks.
You complain about my testing methodology which uses an actual VM load to generate the load rather than using fio to generate a synthetic load whereby you can't tell me the circumstances under which any VM would be able to impart this kind of a synthetic load and then you talk about real systems in real deployments, but then use a synthetic load (rather than a real) load to run your tests.

How are you so internally inconsistent with yourself?

If you're going to talk so much about how bad gluster sucks, in the real world, in real servers, running real applications, then why on Earth would you run a [ib]synthetic[/b] load?

That makes absolutely no sense, given what you've been complaining about.

You'd think that if you were complaining about how bad gluster sucks, in the real world, in real servers, running real applications, that you'd be running real workloads to prove your point.

But instead of doing that (which would be at least consistent with everything you've been complaining about up to this point), you choose to run a synthetic workload instead, and quite possibly your most internally inconsistent and one of your most baffling choice that you've made thus far.

But hey - you do you.

If you need a synthetic load, and one where you can't even explain nor describe how you would even actually get/put a VM into such a state -- more power to you.

What this actually proves/demonstrates is the extremely low probability of your benchmarking results = reality due to the extremely low probability that you'd even be able to put a VM into such a state, in order to get the synthetic load that you were using for your benchmarking.

And the fact that you can even explain/describe the commands that you'd have to run to even get/put a VM into such a state tells me how improbable the situation/scenario you've manufactured really is.

In other words, you had to make up a scenario to generate the data/numbers that shows gluster performing somehwere 75% of cephfs and ~80% of ceph_rbd, and even then, it wasn't like ceph was stomping all over gluster, despite your synthetic workload running on hardware (and software) that you're too embarrassed to disclose what you're using/running your tests on/with (as you complain about the fact that I'm running on 11-13 year old hardware).

Therefore; what this also means is that in the absence of this contrived, synthetic workload, gluster would perform probably just fine, if not slightly above (or slightly below) cephfs and/or ceph_rbd.

This is what your data and your methodology actually shows, given that you can't tell me the commands that I would need to run to put a VM into this kind of a state where it has a queue depth of 32, and trying to read 128 files simultaneously (across 4 VMs at that, where the probability of all 4 VMs running this exact same workload, in any actual real, practical deployment would be virtually impossible).

This is what your data actually proves/shows.
whereas Ceph uses CRUSH to avoid talking to unnecessary nodes.
Again, what you're really saying here is that the ceph devs are wrong.

Screenshot_2026-06-03_06-59-45.png

I mean the ceph devs explicitly tell you "Current code rads all data chunks" (and the picture shows you that it reads all of said data chunks from all OSDs.

You can argue against the ceph devs and say that said ceph devs are wrong all you want, but at the end of the day, if when it comes down to whom I would trust to know more about ceph if it is between you and the ceph devs, I'd pick the ceph devs all day, any day of the week and twice on Sundays.

(I find it hilariously laughable that you seem to think that you know more about ceph and how ceph works than the ceph devs. lol....)

Again, you only need to spend a few hours with ceph to realise that that EC ceph has a synchronisation problem. It's so obvious. And the fact that the ceph devs tells you. (cf. https://docs.ceph.com/en/latest/dev...nhancements/#current-overwrite-implementation, https://docs.ceph.com/en/latest/dev/osd_internals/erasure_coding/enhancements/#direct-read-i-o).

(It is quite evident that you haven't read the ceph erasure coding enhancements document that the devs have already put together. You should probably read that. A fair of what I've said and written about has already been documented by said ceph devs and is a known performance issue due to how the functons/operations are implemented. Will it stop or prevent you from arguing with the ceph devs? Probably not (on account of you still think that you know how ceph works better than said ceph devs).)
The synchronization issue you mentioned is real on Gluster, more bricks, more traffic
As the ceph devs have explicitly documented, this is true for ceph OSDs as well. See the ceph erasure coding enhancements document linked above.

Again, gluster realised and figured this out quite some time ago, when they advised/recommended people make gluster bricks out of ZFS pools because your ZFS pools can be whatever size/how ever number of disks you want there to be (in said ZFS pool) but then you present said ZFS pool (consisting of x number of disks) to gluster, as one brick.

That way, gluster only has to manage the one monolithic brick, rather than the individual disks=bricks and let ZFS deal with the brick-to-disk "translation" of sorts.

That is literally how hybrid OpenMP/MPI works for LS-DYNA. Intranodal communication is handled via OpenMP (e.g. ZFS in this case) and internodal communication is handled via MPI (gluster in this case).

Ceph hasn't quite gotten there yet.

But I've already written about and proposed basically the same idea/strategy so that ceph isn't dealing with 300+ OSDs. You make a ZFS pool out of disks and then you export that to ceph as a monolithic OSD so that ceph only has to deal with the OSD at the ZFS pool level, rather than at the individual disk level.

I came upon this very same idea when I found out that cache tiering has been deprecated in ceph, and therefore; if you use a ZFS pool as your OSD, then you'd be able to get at least some form of cache tiering by way of ARC, to help speed up reads, at minimum.

This is such an obvious solution to such an obvious problem -- a solution that gluster has recommended presumably years ago when they advised in favour of using ZFS pools as bricks.
I didn’t test Windows because it doesn’t matter, in Linux it is easier to control all the variables and automate the tests.
You could've literally just downloaded CrystalDiskMark from within the Windows VM and click "run".

It's not that hard. Since when did downloading CrystalDiskMark and clicking "run" become rocket science for some?
The fact you end up with lost data rather than a complete halting state is one of the many design issues with Gluster and hence the recommendation not to use it in production. The fact it, even if it were a regression, hasn’t been fixed in over 2 years just demonstrates it is a dead project.
You state/argue that it hasn't been fixed "in over 2 years", and yet, if you actually read the thread (which I can only surmise that you didn't), Ravishankar N at Redhat literally wrote:

"There was a case of AFR reporting spurious split-brain errors but that
was fixed long back (http://review.gluster.org/16362) and seems to be
present in glusterf-4.1.6."

Furthermore, i you read Erik Jacobson's reply to him, Erik writes:

"The strange thing, as I mentioned, is it only happened under the job
launch workload. The nfs boot workload, which is also very stressful,
ran clean with one brick down."

...

"> When you say accessing the file from the compute nodes afterwards works
> fine, it is still with that one server (brick) down?

I can no longer check this system personally but as I recall when we
fixed the ethernet problem, all seemed well. I don't have a better
answer for this one than that."

Thus, based on what Erik wrote, sounds like it was an ethernet problem, and not a problem with gluster, as you like to claim.

Which again, I can only surmise that you thought that it was a gluster problem because you didn't bother to read the rest of the message thread. (Since Erik tells you what the fix was, explicitly, from said message thread.)

But you're never going to admit that you didn't read the actual message thread, are you?

You got your confirmation bias and then skipped reading the rest of the thread, where Erik explicitly tells you how they fixed it/what their root cause of the problem was. (Hint: it wasn't gluster)

But, for you -- nevermind that. You got your confirmation bias and stopped reading after that.
 
Yes. dev and support have time and cost associated. In a world where both are at a premium, responsible devs direct those scarce resources to where they have the most value. I would have made the same decision, but good on you for shouldering the slack.
Stupid question though - if Proxmox already has the glusterfs storage plugin that allows it to work from the PVE GUI, then why would that need to change?

I'm not sure that I follow.

I mean, you guys already have it.

It already works.

And as far as I can tell, when you create a VM, epsecially from within the PVE GUI, it asks you where you want to put the disks, and so long as the gluster storage plugin already exists, then why can't you just continue to use that, "as-is"?

I mean, if we translate "gluster is dead" as "gluster is stable" (hence lack of movement on the project), then why would you need to do anything with the gluster plugin?

I'd have to imagine that nfs-common is probably pretty stable with very few commits as well (why break the NFS common client when it works?) and the PVE plugin for that still exists, no?

I'm not sure if this is the nfs source code that's for the nfs-common package, but there are 0 commits over the past year.

https://github.com/linux-nfs/nfsd/graphs/commit-activity

Again, if the gluster plugin exists already and it works, why change/remove it? As I recall, VM creation with Proxmox just asks where you want to point the vm disk to and qemu/qm , as far as I can tell (and maybe I'm wrong), it just wants to know where to put the disk (on a volume backend). qm doesn't manage the storage backend. pvesm is responsible for that, which is separate from qm and thus, QEMU right?

So....if pvesm sets up the volume backend, and qm only consumes or utilises said volume backend, this would mean that QEMU isn't managing the storage/volume backend anymore, right?

And given that the gluster plugin already exists and works, if qm and/or QEMU doesn't manage the storage/volume backend, or that QEMU, the way that it is implemented in Proxmox, doesn't need to talk directly to gluster, therefore; gluster being deprecated/removed from QEMU would be irrelevant to how pvesm works, right?

Or am I missing something?

Therefore; if gluster is handed by pvesm rather than by QEMU, and the gluster plugin already works, then why would Proxmox need to remove the independent gluster plugin, if pvesm and qm/QEMU function/operate separately and independently of each other?

I mean, yes, qm create still expects a place to put the VM disk on a volume backend somewhere but as far as qm is concerned, it doesn't really care what type/class the storage/volume backend is no? So long that you point it to a storage/volume backend, right?

From qm(1);

Code:
qm create
...
--sata[n] [file=]<volume> [,aio=<native|threads|io_uring>] [,backup=<1|0>] [,bps=<bps>] [,bps_max_length=<seconds>] [,bps_rd=<bps>] [,bps_rd_max_length=<seconds>] [,bps_wr=<bps>] [,bps_wr_max_length=<seconds>] [,cache=<enum>] [,detect_zeroes=<1|0>] [,discard=<ignore|on>] [,format=<enum>] [,import-from=<source volume>] [,iops=<iops>] [,iops_max=<iops>] [,iops_max_length=<seconds>] [,iops_rd=<iops>] [,iops_rd_max=<iops>] [,iops_rd_max_length=<seconds>] [,iops_wr=<iops>] [,iops_wr_max=<iops>] [,iops_wr_max_length=<seconds>] [,mbps=<mbps>] [,mbps_max=<mbps>] [,mbps_rd=<mbps>] [,mbps_rd_max=<mbps>] [,mbps_wr=<mbps>] [,mbps_wr_max=<mbps>] [,media=<cdrom|disk>] [,replicate=<1|0>] [,rerror=<ignore|report|stop>] [,serial=<serial>] [,shared=<1|0>] [,size=<DiskSize>] [,snapshot=<1|0>] [,ssd=<1|0>] [,werror=<enum>] [,wwn=<wwn>]
Use volume as SATA hard disk or CD-ROM (n is 0 to 5). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the [I]import-from[/I] parameter to import from an existing volume.

cf. pvesm(1).
 
Last edited:
A cluster file system should remain stable, even in case of hardware failure. I did read that thread and the point is that there was data unavailability that simply allows the OS to continue in a corrupted state. They didn’t know why only that it happens under load. The point of a cluster file system is that it continues working flawlessly even if you pull a network cable or an entire node down.

Gluster is being removed from the QEMU project. This is primarily because there are no developers able or willing to work on it, not because it is no longer a functional system for today’s world, there are other things in qemu that are ancient, like e1000 and even netware era NIC, as long as it works or someone maintains it.

It was a product of its time, as I said, I worked with it for a few years back in the day. Plugins don’t just drop into the system, you seem very confused about how these things work in general but no, the plugin is not just writing files to an NFS/FUSE mount. It leverages Gluster Block support. As to the rest of your comment, you’re reading things that aren’t there, it’s called the Dunning-Kruger effect. You should understand what the Ceph developer writes when it comes to CRUSH or ask ChatGPT to explain how it scales vs AFR/DHT-based systems.

My point on Windows is that you cannot reliably benchmark it if you don’t understand the underlying architecture. It doesn’t matter what your benchmark shows because you’re not testing the storage layer, but the CPU and RAM. I have controlled for those in my test, to do so on Windows would require significant work.
 
Last edited:
This will change as soon as qemu remove it's gluster support.
Again, unless I'm missing something here - is the pvesm a part of QEMU???

And maintaining a qemu fork is a whole different beast than continueing to ship a storage plugin.
But as I understand it, qm that PVE uses, doesn't set up nor configure nor "talk" to the storage backend directly, right?

Again, unless I'm missing something here.

All qm expects is for you to point to a location where it can dump/write the vm disk file and as far as I can tell, it doesn't particularly care what the storage backend that you're using, for the volume is right?

I mean, this is why you have all of the various different storage options, that pvesm manages/configures for qm to use, correct?

Therefore; so long as the management and configuration of the storage backend is separate from qm(1), who cares is QEMU is dropping support for gluster?

As far as I can tell, PVE's qm(1) isn't and doesn't need to manage/configure/"talk" to the storage backend directly (because even using the gluster plugin, in the current state right now, will just mount it to /mnt/pve/<<name_of_gluster (which you can either define/add it via the GUI and/or by modifying /etc/pve/storage.conf), right?

Like I can understand it if qm(1) is likewise, used to manage/configure/talk to "gluster" (as a gluster client) directly, then sure.

But given that you have pvesm which handles the management and configuration of the storage backends, which means that qm doesn't do that, then who cares if QEMU is dropping support for gluster?

qm isn't using it anyways.

pvesm is what deals with the different types of storage backends anyways.

Again, per the qm(1) man page, volume is just the where you're putting the vm disk, not how (you're putting said vm disk).

Code:
Use volume as SATA hard disk or CD-ROM (n is 0 to 5). Use the special syntax STORAGE_ID:SIZE_IN_GiB to allocate a new volume. Use STORAGE_ID:0 and the import-from parameter to import from an existing volume.

So long as the two are separate (qm creates/manages VMs and pvesm manages and configures the different tyles of storage backends, then it would be irrelevant that QEMU is dropping gluster support because pvesm mounts the storageID to /mnt/pve/<<storageID>> anyways, which means that effectively, nearly all storage backend types are all mounted (and exposed) to qm as a directory/mount point.

Again, unless I'm missing something - this is how it works, no? That pvesm and qm are separate, therefore; QEMU dropping support is irrelvent because pvesm exists?
 
A cluster file system should remain stable, even in case of hardware failure.
How do the heck do you get a split-brain with It is a Distributed-Replicate volume 3x9. gvol???

It's a (distributed) replicate gvol.

Therefore; copies of the data are replicated across the bricks, so how would you even get split-brain with a replicated volume?

I don't understand how that's possible.

Furthermore, Erik wrote that it's a distributed-replicate volume 3x9, but from his gluster volume info, the output of that shows:

Code:
Volume Name: cm_shared
Type: Distributed-Replicate
Volume ID: e7f2796b-7a94-41ab-a07d-bdce4900c731
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
I did read that thread and the point is that there was data unavailability that simply allows the OS to continue in a corrupted state. They didn’t know why only that it happens under load. The point of a cluster file system is that it continues working flawlessly even if you pull a network cable or an entire node down.
lol.....you're clearly not on the ceph mailing list, are you?

People have been reporting all sorts of issues with ceph.

ceph isn't immune to issues, just because it's ceph.

I've been on the ceph mailing list since March 9th, and in just the last three months, that folder, in my email, has 221 emails in there, where the issues are quite diverse.

You seem to present ceph as though issues like these only happen with gluster and that issues doesn't happen with ceph, when, as the mailing list shows, is far from the realities of ceph deployments where ceph isn't immune to data unavailability as well. This isn't just a gluster issue. It can happen with ceph as well.

(cf. here, here, and here as well.)
I did read that thread
If you read the thread, then you would've read that Erik stated that they fixed the problem that had nothing to do with gluster itself anymore than if you have a bad drive capable, which causes data corruption, would be the cause of whatever HW RAID system you were using, at the time.

Again, as I've cited, ZFS will commit corrupted data to stable storage if you have bad drive cables and/or issues with the IMC where ZFS won't know that it is commiting corrupted data to disk.

That'd like you blaiming gluster = bad because of a bad network cable, where if you fixed said bad network cable, then your gluster problem goes away.

Cuz that's what Erik from HPE wrote, which if you read that, then you would've known this rather than saying, quote;

"Similar split brain scenarios under heavy (note that for today's standards, this is no longer heavy, Ceph can hit 1M 64kB IOPS across 12 nodes) load was reported before on mailing lists in 2019 with Gluster 4 https://lists.gluster.org/pipermail//gluster-users/2019-September/037083.html It is a very similar problem to what I started seeing 15 years ago as our cluster grew, split brains started occurring. Yes, you can throw more CPU, hardware and do optimizations, but at some point, you get to a point that it is untenable, a file system that throws up its hands in I/O error is BAD."

Erik, in his reply to Ravishankar N at Redhat, literally tells you this.

But that's not what you wrote in the quote from you above.

If you read the thread and knew it, then that's knowingly making fraudulent claims/statements about gluster that you [i[know[/i] were fraudulent misrepresentations.

Therefore; given what you wrote, I would have to give you the benefit of the doubt where you didn't read said thread.

So which is it? You read the thread and then made knowingly fraudulent misrepresentations anyways, or you didn't read the thread, which is why you wrote what you wrote?

So which is it?
This is primarily because there are no developers able or willing to work on it
From the commentary from others, that's not the claimed reason why.

The claimed reason why is that they were looking at the relative lack of activity, but contrary to people's claims (that are then promulgated and repeated elsewhere like here), gluster has active developers even if the pace of commits isn't as active as a project that has a lot of problems that i needs to patch.

I mean, that's the easiest way to boost the number of commits per month. Have (or introduce) a lot of bugs that you have to fix. You can push the commit/month metric through the roof when you have a lot of bugs to fix.

(cf. Nvidia drivers)


that are ancient, like e1000 and even netware era NIC, as long as it works or someone maintains it.
Just out of curiosity - how many commits have there been for the e1000 NIC driver within any timeframe?


Plugins don’t just drop into the system, you seem very confused about how these things work in general but no, the plugin is not just writing files to an NFS/FUSE mount. It leverages Gluster Block support.
Again, as it has been noted, you can mount the glustefs with the glusterfs-client package (in the case of Proxmox, from the Debian repo). Therefore; so long as the GUI/plugin is to provide a graphical front end for said glusterfs-client package, which already exists on Debian, and as @kayson mentioned, Debian's newest package supports gluster 11.2, therefore; so long as Debian supports it, the plugin can just be the graphical front end that sets up or passes the user input arguments to the glusterfs-client to mount a gluster volume.

Said gluster block support is already provided via glusterfs-client.

You think that I'm confused and yet, you haven't been able to provide any citations that shows that I'm wrong (about how the glusterfs-client works).


As to the rest of your comment, you’re reading things that aren’t there, it’s called the Dunning-Kruger effect.
More often than not, the people who cite Dunning-Kruger completely fail to realise that there's an entire right-side of the Dunning-Kruger effect plot that they often times, generally and convenient ignore.

(Ironically enough, and since you mention it -- you're still on the left side of your citation of the Dunning-Kruger effect where you think you know more about ceph than the ceph devs. This is a literal and practical demonstration of your reference of said Dunning-Kruger effect where you don't even know what you don't know.)

Screenshot 2026-06-07 005813.png

It's literally right there.

LOL...LMAO....

I can't help if you're unable to read (or too lazy to read -- can't really tell at this point whether it's a question of ability or whether it's a question of laziness and/or both.)


You should understand what the Ceph developer writes when it comes to CRUSH or ask ChatGPT to explain how it scales vs AFR/DHT-based systems.
Or...you can just read what I've drawn the red box around from the ceph documentation that the ceph devs have authored.


My point on Windows is that you cannot reliably benchmark it if you don’t understand the underlying architecture. It doesn’t matter what your benchmark shows because you’re not testing the storage layer, but the CPU and RAM.
1) Translation: you're too embarrassed to disclose what hardware/software you're using. Which is fine, because after all your complaining about it, what this means is that the hardware that you're running in your homelab would be susceptible to your own complaints about said hardware which is why you're embarrassed to disclose what hardware/software you're using.

2) You're embarrassed to have someone else replicate your tests because if someone else dares tries and am able to generate data that contradict your claims -- well, you just can't have that.

3) You will critique what WIndows/NTFS does all day long, but then do nothing about "fixing" it (because in reality, the vast majority of Windows users doesn't GAF. And by market share, Windows is still the dominant desktop OS, by far, which also means that if the issues that you've raised were really as serious as you make them sound, they would've been fixed a long time ago, because too many people would have lost their data because of how Windows/NTFS behaves. And yet, that's clearly not the case here.

Millions/billions of people have been using Windows just fine just as millions/billions of users don't care about the points that you've brought up (about how Windows/NTFS quote "lies" to you given that it works "well enough" for said millions/billions of users.

So, it's a whole lot of nothing burger.

4) You can't prescribe/produce a script where you'd be able to put a VM in such a state to generate the workload that you're using fio to test with/generate, even though you claim that Windows can do it.

So, if Wndows is able to generate/produce that kind of a workload, then why bother using fio???

Heck, use Linux for all I care (given your complaining about Windows).

According to you, any modern OS would be able to generate that kind of a workload, so then why don't you just use quote "any modern OS" for the load generation rather than fio if you claim that any modern OS can generate that kind of a load???

If it can do it, then why aren't you testing it with it then??? Seems so simple.

And for the record, I re-ran CrystalDiskMark 24 hours and 40 minutes later and this is what I got;

Screenshot_2026-06-04_05-52-36.png

You argue that it can't be tested reliably and yet, the maximum stdev (in MB/s) was 4.13657 MB/s and the min stdev was 0 MB/s. As a percentage of the average stdev/mean was 2.28% for gluster and 3.94% for ceph (because it was so slow, that small differences in the results yielded a higher stdev, and therefore; as a percent of the average, the slow results of ceph made it more sensitive to variations.

Either way, I could run it over and over again to obtain the population statistics, but the relatively low stdev already shows that it will pass a gauge R&R.


I have controlled for those in my test, to do so on Windows would require significant work.
What you see on the host side isn't necessarily what you're going to see, from inside a VM.

You argue that you had to cap the network at 400 Mbps meanwhile I installed 3 Win11 VMs, on the gluster cluster, without any issues. (System was RAM limited such that I couldn't spawn a 4th gluster node, and then spawn a 4th Win11 VM.)

You said that it would split-brain under quote "a heavy workload" (which is why you tested with fio --iodepth=32 --numjobs=4 --nrfiles=128) and yet running three Win11 installs ran without any hitches and I didn't bother neutering the virtio-nic so that it will be able to run at full speed.

In other words, your synthetic workload is precisely that - synthetic. It isn't a "real" workload of Windows (or Linux since complainers have to always complain about something) that would be more realistic and more representative of what an actual client would be running or can run.

Again, I don't need any tricks nor do anything special/extra.

Just install and go let the post-install Win11 updates do Win11 update things.

You could automate the install and testing with DISM.

I ran my testing/benchmarking the way that I have because it is designed to be so simple that anybody could repeat the tests since there's hardly anything to it.

Point your VM to the Win11 ISO and run through the process of creating said VM and then once it is all set up, just let Win11 updates to Win11 update things.

This is performed this way, because this is what Win11 installs are like. Nobody bothers customising anything (in terms of how NTFS reports drive statistics back to the user because no one GAF).

It's what people do.

And by purposely running the tests this way, it means that any can repeat the test, just as I didn't bother tuning ceph/gluster neither.

Install and go.

This is what the experience is when you just deploy ceph/gluster/Win11 VM "out of the box". This is what you get.