Ceph in 2026 HDD

[guruevi]

@J-Rod 300TB for $100k? When? A loose SAS 30TB disk from retailers if they are even in stock are selling out at $6k. Dell told me 18 months if I ordered in April for a PowerScale system with 24TB and the price had tripled to $1.2M/PB. What’s available today in limited quantities is ~18TB. SSD are somewhat more available.

 

[J-Rod]

I don't remember what the delivery date was when I quoted it previously, but today it states October 15th.

Seems normal for the higher capacity drives in a new product series to be excessively priced and possibly in short supply. There is a sweet spot of capacity/$ and availability.

Side rant:
I don't subscribe to the industry-standard, blind product replacement cycles. Companies perpetually want your money, so they are going to be aggressive about dropping support and also coming out with new features to entice you to buy it all over again. I'm running PVE on a 15+ year old server with PBS as a VM for archival/DR backups and there is effectively little difference between it and something brand new. Power draw is about 30% higher but this means little since it is only on a couple hours a week. Even with 24x7 operation a new box would take decades to break even. It will be replaced with a 10 year old box from production at some point I suppose.

 

[alexskysilk]

I'm running PVE on a 15+ year old server with PBS as a VM for archival/DR backups and there is effectively little difference between it and something brand new.

you may not want to look at a killawatt then. a 1L box will provide roughly equivalent performance at 1/5th the power draw- and thats not even brand new. 15 year old server = 150W+ at idle.

 

[J-Rod]

I have charting for approximately 20 outlets/devices so I know where all the energy in the house is going. With all 8 HDDs spun up, this system sits at 90W, but as mentioned it is only on a couple hours a week (and actually, all of my backup and archive servers turn on daily/weekly when solar power production is at its peak). Why spend any money on it when I can use that money for production systems that are on 24x7? And even if it were on 24x7, to buy something newer but still used could take decades to break even. Also, for this server I specifically mentioned archival & DR use because of course there are situations where performance per watt is worth considering, but in this case it's just not.

My core production pulls about 375W continuously under typical load, and that is a 3xPVE cluster with 7xSSDs per node, plus three 10G switches and two all-SSD NAS units. This is absolutely all older gear and I regularly look at what could be replaced with new or newer, for how much and how far out the break even point would be. Even before the current pricing bubble, it just never penciled out, not even close. It's at least 4X cheaper to just keep running the equipment I have now and put the money into my retirement account.

Really the main point I was trying to make is that you can keep older equipment around and not have to blindly replace things every five years like the industry "demands". Just be smart about it. Here's a real-world success story of a company that went from cloud to running their own servers and keeping older R630 servers was part of that: https://world.hey.com/dhh/servers-can-last-a-long-time-165c955c

 

[guruevi]

What you describe is good enough for a home lab, but not in the real world. First of all, warranties and replacement parts are a thing. Besides that, your server will suck down more money in power and cooling and licensing than it actually costs in hardware. If you have a server that sits idle (what you describe), use a Raspberry Pi or other low power ARM device. In a datacenter, every kW costs triple (once to power, twice to cool).

Even Proxmox, $2200/server * 5 years = 11k/server. If 5 years from now, you can run 4x more servers or run the same load with less cores, per-U costs etc, that is an instant savings, not to speak about the real software costs, the per-VM Ubuntu Pro, RHEL subscriptions, Windows licenses etc.

Hence we buy new servers not because the vendors tell us to, but because it is cheaper to buy 5 new servers than to keep 20 old servers up and running.

As to the article, a quick back of the napkin calculation shows they could consolidate on less servers and assuming their workload stays steady and they pay average costs for power and colocation, they could save >$100k in the next 5 years by replacing, even accounting for depreciation. This works even better financially if they spread the purchase over 5 years and rotate in newer hardware. And with that kind of scale, you may eventually reduce your labor costs as well.

 

[J-Rod]

Actually, I've been in the "real world" for 30+ years. And I never stated to not buy ANYTHING new, just be smart about it...

Warranties and support contracts are huge money makers for manufacturers and it's cheaper to just buy a percentage of extra systems or get parts from any number of reputable third party sellers. Also, we know how warranty claims work: deny, deny, deny. Or it's out of stock, or no longer available from our "supplier", etc. Anything to get out of it.

Power and cooling costs are variable, and in many areas it's just not expensive enough to balance against outright replacing working gear after five years, especially for general use workloads.

For licensing, you selected the most expensive Proxmox license of 1,100EUR/socket/year, whereas anyone who is concerned about cost would more likely select basic or standard, or only purchase Premium for the core critical cluster(s). For Linux, same deal, you may buy RHEL for your critical nodes but you'd surely run Rocky everywhere else. Regardless, the extra levels of consolidation that you're assuming at each five year mark is really quite generous.

The 37Signals folks seem to have a good grasp of where their money is going, and it aligns with what I've observed over the decades. There are some clear generational jumps in technology where it makes sense to dump older gear, such as CPUs without VT-x (or d) support. The 768GB of RAM and decent core count probably factored into their decision to keep the R630s. A stack of R630s with 128GB RAM and 8-core CPUs would likely be bounced out the door in a heartbeat, in this specific scenario.

 

[guruevi]

I never had problems with warranty coverage from reputable suppliers. Depending on the level you buy, it saves you a trip to the datacenter, which in itself is a huge opportunity cost depending on where you are and how big of an operation you have, not the first time I would have a tech accidentally damage something because they aren’t well trained. Especially with GPU that are on the order of $45k per part, the overhead on warranty is well worth it.

The Proxmox licensing is the least expensive probably, have you seen the VMware or Citrix Xen or Nutanix or Microsoft HyperV licenses. Sadly, if you are on a Microsoft HCI train (HyperV server is practically discontinued and continues under the Azure HCI branding) , they actually charge the same amount for cloud vs onprem (per-hour and per-GB).

Likewise for RHEL vs Rocky, the value is in the support. The support and training costs are where the value lies.

As to the generational differences, compute still doubles every 18 months. Having a 4x increase after 5y is not unlikely. Again, if you over-purchased 5y ago, that just goes to show that people don’t know how to scale properly. If 65 nodes do the same work as they did 10 years ago, either your business stagnated or you wasted a ton of money 10 years ago, neither of those point at great leadership.

 

[alexskysilk]

You guys are arguing from two fundamentally different perspectives.

Can you continue to use fully depreciated servers? sure. it also has the benefit of (relatively) fixed opex. When you're a "homelab" or small business, the overall expense of the upgrade can and often is more "expensive" of just leaving the whole infrastructure alone and produces no additional utility.

From a "revenue lifeblood" perspective the variables are different. as @guruevi pointed out, trading licensing/power/cooling costs for more efficient hardware, maintainability at scale, and the sheer financial mechanics of depreciation make upgrades not just worth it but required for good stewardship.

Different considerations for different use cases.

 

[guruevi]

My argument goes farther than that. As I pointed earlier, if a 10-15 year old server at near zero load can do what you need today, sure it’s fun to play with but beyond that, there is an argument to be made that ARM devices today are much more power efficient and just as good for those use cases. This has an immediate effect on people’s energy costs.

In Europe, where people now pay 25-50c/kWh (and rising) and see massive power shortages, even a 100W server costs ~$200-400/year, ‘too much’ for many home-labs, a 10W ARM-based system is quickly paid back over a 7-10 years lifespan, even accounting for the relatively small (especially second-hand from e-waste companies) price difference between a few 4TB SSD and HDD. Every 10W you save (eg eliminating an HDD for an SSD) is $200+ over that 10 years.

Sure, if you live in the US, and like me, pay 4c/kWh, that argument goes to the wayside, and my home lab is ‘all free’ from the waste pile, with a 1kW bitcoin mining rig acting as my space heater, but I could not justify that for a client (old hardware just tends to die more often, support call is $150/h). If you can use second hands today, you likely could do so a long time ago too, and again, you get in the argument of over-buying (why buy dual-Xeon if all you need is an Atom). I now buy $300 Atom systems for customers that just need a simple router, the cost of refurbishing some older ‘enterprise’ system is just too high. And even my home lab itself has shrunk over the last few decades as my hardware gets ‘younger’.

 

[Johannes S]

As I pointed earlier, if a 10-15 year old server at near zero load can do what you need today, sure it’s fun to play with but beyond that, there is an argument to be made that ARM devices today are much more power efficient and just as good for those use cases. This has an immediate effect on people’s energy costs.

ProxmoxVE hasn't an official ARM port though, I agree with the rest of your points though. And Incus is available for ARM.

 

[alexskysilk]

there is an argument to be made that ARM devices today are much more power efficient and just as good for those use cases

X86 hasnt been static. performance/watt is similar in MOST current arm and x86 examples with the overt exception of Apple hardware that knocks everyone out of the park- but for @J-Rod use case it doesnt matter because he keeps his offline when not in use.

 

[J-Rod]

I've tried to be clear that my position revolved around "blind" replacements, so if five or even three years pencils out, then more power to you. And again, not every workload is just compute, so we have need of memory, storage, backups, DR, even perhaps cold spare capacity because things can be dynamic (Covid, AI bubble).

My local ISP doesn't charge for power usage, yet electricity here is nearly 40c/kWh. Minimum buy is one rack for one year.

A couple local companies I support use their rack space only for backup and DR due to connectivity limitations, and the sites need to keep running continuously if connectivity is lost, hence on-prem equipment. They shuffle the older gear down from primary production to the DR site, and about 10 years is the total lifespan of the hardware in this specific situation.

At the 8-5 job, we do get charged for power usage, by our own company. So we leverage power management solutions and power down compute nodes during longer idle periods. We recently repurposed some compute nodes for an OpenStack HCI cluster and power draw is just not really much different from brand new, because the cluster is mostly about memory capacity and I/O.

Low power isn't a new development. It just wasn't always a primary focus for enterprise products. I've run NUC clusters since they came out, and before that I used thin clients (Via C3, anyone??). So a handful of watts going back 25+ years to handle lots of workloads for myself and many customers.

Now leave me alone, because I have a pile of 15th gen Dell servers in the garage that were recently tossed out and I'm trying to think of what to do with all of them.