480GB Pcie3 is of no use today in homelabs.
SATA SSDs are of no use today in homelabs.
Proxmox is destroying my NVMEs
- Thread starter manguy
- Start date
Well, I wanted something new and reliable, with a long lifespan. Almost 10 years ago, I built my main server using new workstation-class components available at the time: a Xeon E3 1240 v5 with 64 GB of ECC RAM and SATA SSDs. I also built a backup NAS based on the Xeon E3 platform with spinning disks. Used (rack) server gear was never an option for me because it's too loud and power-hungry. And back then, enterprise drives were also considerably more expensive than they became in the years that followed, so I went with new consumer drives.
The main server is a tower build with a Supermicro motherboard, Seasonic PSU, 2x 250GB and 6x 1 TB Samsung EVO SATA SSDs. It cost me quite a bit back then, but IMO it was worth it. It's stll running perfectly fine, and the boot SSDs are now at around 40–50% wear.
My plan was to build a replacement server with a similar budget to what I had back then, but with more ECC RAM and CPU cores. I would use current-generation hardware and perhaps even enterprise-class U.2 or U.3 drives. Obviously, I would use fewer, larger drives this time. The idea was to build something that would last another ten years.
But guess what? Even if I could find all the components I need, the build would now cost roughly two to three times as much as it would have cost a year earlier, putting it well into five figures. So, I then bought a 1L mini PC with 64 GB of RAM to expand my setup without replacing the main server. That was a the beginning of this year, when you could still get a PC wit 64GB of RAM for a three-digit price,
It came with 2x 1TB WD Black SN7100 NVMe SSDs, which are working just fine so far (still on 1% wear) and feel noticeably snappier than the SATA drives in my main server. Random I/O is probably more important here than raw throughput. The DC2000 would likely perform better in that regard than SATA SSDs as well, and perhaps even better than the WD Blacks. But at roughly three times the price of a new WD Black, that's a pretty hard pill to swallow.
I mean, enterprise SSDs have always been more expensive, but now that everything is in a completely different price bracket, they're no longer a realistic option for most people. I mean just look at the 1TB NVMs: Paying €200 instead of €80 is one thing, even though that's already three times as much. Paying €586 instead of €200 is quite another. And just under €100 for a used 480 GB drive isn't exactly appealing either, especially if you remember what prices were like two years ago.
It's all a bit frustrating, to be honest. But anyway, those used DC1000 drives are certainly not a bad deal in this time and age, and I might even consider going that route for the mini PC if the WD Blacks eventually fail. Although, if things keep going this way, by the time that happens, used 1 TB enterprise drives might cost more like €200-300, assuming you can find any at all while new 1 TB consumer SSDs might cost €300–400.
The main server is a tower build with a Supermicro motherboard, Seasonic PSU, 2x 250GB and 6x 1 TB Samsung EVO SATA SSDs. It cost me quite a bit back then, but IMO it was worth it. It's stll running perfectly fine, and the boot SSDs are now at around 40–50% wear.
My plan was to build a replacement server with a similar budget to what I had back then, but with more ECC RAM and CPU cores. I would use current-generation hardware and perhaps even enterprise-class U.2 or U.3 drives. Obviously, I would use fewer, larger drives this time. The idea was to build something that would last another ten years.
But guess what? Even if I could find all the components I need, the build would now cost roughly two to three times as much as it would have cost a year earlier, putting it well into five figures. So, I then bought a 1L mini PC with 64 GB of RAM to expand my setup without replacing the main server. That was a the beginning of this year, when you could still get a PC wit 64GB of RAM for a three-digit price,
It came with 2x 1TB WD Black SN7100 NVMe SSDs, which are working just fine so far (still on 1% wear) and feel noticeably snappier than the SATA drives in my main server. Random I/O is probably more important here than raw throughput. The DC2000 would likely perform better in that regard than SATA SSDs as well, and perhaps even better than the WD Blacks. But at roughly three times the price of a new WD Black, that's a pretty hard pill to swallow.
I mean, enterprise SSDs have always been more expensive, but now that everything is in a completely different price bracket, they're no longer a realistic option for most people. I mean just look at the 1TB NVMs: Paying €200 instead of €80 is one thing, even though that's already three times as much. Paying €586 instead of €200 is quite another. And just under €100 for a used 480 GB drive isn't exactly appealing either, especially if you remember what prices were like two years ago.
It's all a bit frustrating, to be honest. But anyway, those used DC1000 drives are certainly not a bad deal in this time and age, and I might even consider going that route for the mini PC if the WD Blacks eventually fail. Although, if things keep going this way, by the time that happens, used 1 TB enterprise drives might cost more like €200-300, assuming you can find any at all while new 1 TB consumer SSDs might cost €300–400.
Last edited:
It definitely hasn't been that hard to find the used enterprise gear! Yes, for some countries you may have to import. But manufacturers like Dell (I'm most familiar with them) have had a consistent product lineup for decades that has included lower-power/lower-noise/budget-friendly/homelab-friendly mini-tower workgroup servers as well as reduced-depth single-socket rack mount servers. For example, the PowerEdge T110/T110ii/T30/T40 and the R210 through R250.
I have seen deals via at least two methods: 1) Dell is refreshing the product line, so the outgoing models are on fire-sale discount; 2) 3-5 year old off-lease or otherwise used gear reaches the reseller market in quantity, leading to an oversupply and low prices.
Currently my 24x7x365 home production cluster is basic. I have three T30 mini-towers with the Xeon E3-1275 v5, 64GB ECC RAM, 2x10GbE and a handful of Dell/Intel SATA SSDs (and sometimes a few PCIe Intel DC P3700). None of the T30 units shipped with that CPU model, that much RAM, the SSDs or the 10GbE cards. So all that I hunted down later via eBay or one my favorite US resellers (https://vibrant.com). This allowed me to snag bare-bones T30 servers for cheap because who wants a server with a dual-core Pentium and 8GB of RAM?
I have seen deals via at least two methods: 1) Dell is refreshing the product line, so the outgoing models are on fire-sale discount; 2) 3-5 year old off-lease or otherwise used gear reaches the reseller market in quantity, leading to an oversupply and low prices.
Currently my 24x7x365 home production cluster is basic. I have three T30 mini-towers with the Xeon E3-1275 v5, 64GB ECC RAM, 2x10GbE and a handful of Dell/Intel SATA SSDs (and sometimes a few PCIe Intel DC P3700). None of the T30 units shipped with that CPU model, that much RAM, the SSDs or the 10GbE cards. So all that I hunted down later via eBay or one my favorite US resellers (https://vibrant.com). This allowed me to snag bare-bones T30 servers for cheap because who wants a server with a dual-core Pentium and 8GB of RAM?
Yeah, I mean, that's perfectly fine. But I wanted current components and a custom build at the time, and yeah, that obviously cost more than a used Dell tower. But the components themselves (motherboard, RAM, CPU, etc.) weren't significantly more expensive than what others spend on components for their gaming PCs every few years. In the end, the storage was the most expensive part.
In addition to the SSDs, the machine also has eight 8 TB HDDs inside for media storage, which is an issue in itself with current disk prices. The oldest of these drives have nearly 80,000 hours on them, and when I consider how much an 8 TB HDD costs these days, I honestly have to wonder whether it's still feasible to keep that much media around. And to expand the storage is something I don't even need to think about at current prices.
But yeah, I guess, we've gone a bit off-topic.
This thread is about how Proxmox supposedly kills SSDs. And based on my ~9-year-old system at least, I would say that even consumer-grade drives can last a long time as long as you don't buy the cheapest ones...
Last edited:
You need to ask what type of writes small unbuffered frequent log updates are? You also need me to explain write amplification of small writes on ZFS to you as well?40% less without stating what 100% was, is not a real number
If it causes writes, tell us how much a day and what size these writes are happening (in case SSD has write amp for small writes).
I get it, if you an enterprise drrive user, or are used to extreme writes, such as enterprise level database loads, this may all seem quite petty to you.
But some of us are not using enterprise drives, we recognise storage is incredibly expensive so dont want our drives to expire early. Or maybe some people just want things to be efficient regardless of the state of hardware. I dont think its right to belittle these concerns, you dont have to worry yourself, but dont belittle those that do.
In terms of pveproxy, the writes were significant enough for me to notice on SMART data and live i/o. Thats all there is to it really. My overall changes when all compounded together including pveproxy, have reduce my background writes by over 90%. Also dont forget that any write triggers a metadata change, so if there is less i/o, there is less chance of meta corruption in case of sudden shutdown. It wasnt excessive effort as I already routinely place nginx in front of pveproxy anyway.
pveproxy by itself was triggering an erase cycle every 8 days, After the changes it increased to around 72 days, much closer to a consumer work load.
Last edited:
I am very happy to admit that I was wrong (because then a learnt something).
That drive has some small capacitors to make sure some SSD tables won't get broken. So you single SSD applicance will always boot. That is their selling point. It doesn't even come with a huge DRAM write cache, heck the disasterously bad 4k sync write benchmarks (at least for a "PLP" drive).
Again, this drive is sold as a cheap boot drive.
I knew I should not have brought up that fake PLP drive into the discussion
Not really.
Kingston 1TB: 5y and 730TBW
Samsung 9100 as 2TB for less money: 5y and 1200TBW
Last edited:
No, but you could provide some numbers for your claims. Because if it is 80k of 4k writes a day, we would both agree that even the cheapest QLC drive could handle that?
why are your drives formatted in 512e mode rather than native 4kn mode. has your ZFS pool also been configured to use 512 bytes too if so your by design wearing out your own disks because you haven't aligned them.
have you 4k aligned your VM's with
qm set <vmid> --args "-global scsi-hd.physical_block_size=4096 -global scsi-hd.logical_block_size=4096"
have you disabled realtime trimming in your guest OS so it's not constantly expanding and shrinking your disks on the storage, only do weekly scheduling.
have you disabled atime 8.3 filename generation etc that cause pointless writes.
you have to take active measures to stop writes and align everything it you want your cheap flash to last as it has no power loss protection.
Last edited: