Yes. ceph requires a minimum configuration (network topology, OSD count, etc) before its performant.
I'd argue the opposite:
If ceph is performant with old, cheap, crappy hardware, then throwing new hardware at it would just make it run better, faster.
(I mean, this is why I am running an entirely virtualised cluster because that way, my GbE won't be the limiting factor for ceph to work the best way that it can. You'd think that using the
virtio-nic, that should mean that as far as networking is concerned, it will be able to run pretty much as fast as said virtio-nic will allow and won't have PHY/hardware/physical limitations like a PHY link would have.
And even
with this, it still can't muster up more than 9.21 MB/s in sequential write on the
same underlying hardware where I'm running my gluster tests.
CPU is the same between both tests, as is the RAM, motherboard, same Proxmox host boot SSD, same HGST 1 TB SATA 3 Gbps HDDs across the board, same everything basically. And the only difference is one virtualised 3-node cluster is running ceph and the other virtualised 3-node cluster is running gluster. And the host, and the virtualised clusters are all running Proxmox 7.4-20, so even down to the software versions, it's all the same.
Again, the video from 45Drives shows that even with vastly better hardware, they're still only able to achieve about 8% of the drive's performance capability (vs. the ~22% that I'm able to currently achieve with this old, crappy hardware).
So now, you take that 22% and apply to new, modern hardware, and so 22% of a U.2 or E1.S EDSFF NVMe 5.0 x4 SSD that's capable of 12 GB/s sequential read/writes will yield 2.64 GB/s vs even the
best of ceph at 8% would only yield 0.96 GB/s.
No amount of networking is going to make up for the fact that EC ceph appears to max out at around 8-ish% of a drive's capabilities whilst a distributed dispersed gvol, even with my old crappy hardware, maxes out at around 22%.
Again, watch the 45Drive video that I've linked above.
When you're going to getting like 0.96 GB/s, what people
actually do, in
actual ceph deployments is that if they want to get closer to the 2.64 GB/s that you might be able to get with gluster, you'd have to buy
three of the U.2 or E1.S EDSFF NVMe 5.0 x4 SSDs to make up for the fact that a ceph doesn't/can't use more than ~8% of the drive's capabilities.
That's literally just throwing money to mask the fact that ceph is only using 8% of the drive. Three times as much money.
vs. if you bought the one drive, and then ran gluster, gluster is faster, in both sequential and random workloads.
Out of all of the participants in this discussion,
no one has provided any real, concrete, hard data that shows/demonstrates/proves otherwise.
People can complain about my methodology because again, as I predicted, people were going to complain about the hardware that I'm using, and since I predicted that people were going to complain about it, (and then the people who are complaining about my hardware of choice isn't
actually going to supply new(er) hardware for me to test with, i.e. they're not going to
actually do anything about it), which either means, if they're going to complain about it, then they can run the very same tests, run it however they see fit, with whatever hardware they have (or have access to), and try to show me that ceph can do better than using
just 8% of a drive's capability.
No one else has provided
any data. They just complain because people who complain won't be people who complain, unless they're always complaining about something (and then do absolutely
nothing about it). It's complaining for the sake of complaining.
its whether you are comfortable with operating a legacy environment without security or bugfixes. Its your headache to deal with. There was/is nothing inherently wrong with PVE7, or gluster- their devs simply moved on due to various drivers.
I think that it really depends on what's your security risk profile and how tolerant you are to/of it.
Like if you're a homelabber and you haven't opened any of your services so that they can be accessed from outside your network, then how big of a security risk/attack vector/profile do you
really have and how much of that are you able to tolerate?
Think about all of the people who have homelabs.
You can find YouTube videos of people who deliberately exposed a new WinXP VM to the internet to become an instant magnet for problems.
But you can also run
dockur/windows, run WinXP, have it still be able to connect to the internet without
any issues. (Ask me how I know this. It was a bit of a pain to try and get a WinXP SP3 CD key.)
So how big of a security/attack vector is it
really?
Think about how many point of sale systems that are
still running WinXP Embedded.
I dont really know what you're referencing in terms of information or statistics
My dad used to work at one of the banks as a "computer operations associate".
Banks do
not deploy bleeding edge technologies. It is one of the reasons why your bank account, probably most likely sits on an IBM mainframe (still) somewhere. There's a very good reason why they haven't migrated your bank account over from AIX/POWER (or z/OS/System z) over to Linux on arm or x86_64. If you've ever noticed how it is becoming more often that you
aren't able to access your bank account online, it's probably because they've migrated -- something that didn't used to happen nearly as much, when it was still running on the IBM mainframes. Or the other way to ask yourself the same question is "who still uses (IBM) mainframes and why?". (Oracle has pretty much functionally and effectively killed of Sun Microsystems hardware. Yes, there are still new servers, but when you go to oracle.com, their hardware is
not featured prominently on their homepage, which tells you everything you need to know about Oracle's perspective on their own hardware solutions.
Where I work, our global manufacturing system, based on what my senior and supervising engineers have told me,
still run on COBOL.
I've also been told that there has been attempts by companies/developers/computer/IT companies that have tried to modernise it and they have all failed to do so properly. If said global manufacturing system stops working, all of our manufacturing plants shut down, as a result/on account of it.
So pretty much any company that isn't a startup in like the last 30-40 years (since the dot com boom), the rest of the companies that have long existed before said dot com boom of the late 90s, are probably running legacy systems that either can't or have failed to be able to modernise (some of which, isn't for the lack of trying).
Or to you put it to you in a slightly different way, when I am interviewing potential developer candidates, I will ask them questions about scale. I had one interviewee who physically flinched when I asked this question (about scale), because that
is the scale of our operations.
We've migrated from an on-prem deployment of github to github-in-the-cloud and I quite literally get emails about
some problem with said github-in-the-cloud service
daily. Again, scale.
Think of any company that existed throughout the 80s and 90s and even earlier. And ask yourself what kind of systems do you think they (still) run?
With regards to IB- its a fantastic low latency transport, and is used in many use cases where latency is king. Unfortunately its also missing a TON of functionality used by more typical usecases (No layer 2 functionality to speak of) which limits it's utility for PVE, for example. Nevertheless, it can be used as long as your comfortable operating SMs in production and dont attempt to use them for vmbrs. I decomissioned all IB from my deployments when 100gbe became nearly as performant but massively cheaper, and with switches that can actually do stuff. I get you already own the hardware so the price isnt a feature for you; the real question you should be asking is what are you actually after?
Three things:
1) As I mentioned, for me, 100 Gbps IB was for HPC that I was running in my basement. As I've also said, said HPC applications that uses MPI/RDMA etc. can regularly and routely hit 80-90 Gbps out of a possible 100 Gbps as the software is leverage the hardware system interconnect.
Also as I've said, storage, as that point, was just a fringe benefit (and also as I've said, I didn't buy said 100 Gbps IB
for storage). I bought it for HPC. The fact that I can use it for storage as well, was just an added bonus.
2) Remember that bandwidth can be represented by bit width / latency. Lower latency
can relate to higher clock speed (but not always). Higher clock speed * bit width loosely gives you bandwidth. Therefore; if you want to speed things up, you can either increase the bit width, increase the clock speed, and/or lower the latency.
(That's more or less how the IB roadmap is developed, where they tackle one or more of these variables, pretty much in terms of signal and/or electrical and electronics engineering.)
3) As I've said, the Linux bridge (which I've come to learn) and it's
inability to create IB linux bridges, isn't just a Proxmox thing; it's a linux thing.
Therefore; given that I
can't create linux bridges, the next best thing for shared IB is SR-IOV and IB VFs, which, again, as I've stated, the Debian version of
opensm doesn't support.
Which is also why I already wrote about how if I want virtualisation support, I'd either need to replace my switch from MSB-7890 to MSB-7800 (which I've had before, but had problems with my previous unit, so I returned it), and/or I'd need to have a dedicated system, whose sole purpose would be to run the IB
opensm, most likely either in CentOS or Rocky Linux, so that I can enable virtualisation support. (which I've also written about already as well).
Then I'd be able to pass the IB VFs to LXCs and VMs and that'll get about as close to a linux bridge as it'll get on the IB side of things.
And the idea with that would be "use it since I already have it" (instead of deploying a new, physical 10G layer). Intranode, it can run 10G internally. That's what the virtio-nic shows up as in Win10+, Linux, and MacOS. Internodal communications can run over 100 Gbps IB (since I already have it).
If you're insistent on using this for whatever purpose- use it. no one will stop you. If you intend to operate it as an enterprise- you've been warned.
Conversely, if an enterprise has already deployed gluster and it's working for them (because it's faster than EC ceph), if it is working for them, then it is working for them.
