Consumer grade SSD's

[TravisH]

Hi All,

Apologies this seems to be a question that comes up a bit, but I wanted to ask about using consumer grade SSD's in a ZFS mirror for home use. In my case I have a spinning disk setup with two 16TB drives in a ZFS mirror arrangement. This seems to work well except that as it is spinning disks the speed leaves a bit to be desired.

I am thinking of possibly having another ZFS mirror of a smaller capacity for the stuff I am currently working on (things like photos I might be editing and the likes) where the speed can be much greater than the spinning disks. I am not a professional photographer so I don't think there would be massive file turnover, but I might have 10 - 100GB each time, and every few weeks (at most) I would be putting these on the SSD.

I see that SSD's can be worn out fairly quickly by ZFS, but I am wondering how much background file movement actually takes place with ZFS that would actually wear them out in a mirror arrangement if I am just uploading files whilst I am working on them, leaving them there for a few months, and then after that moving them to the spinning disk / archive storage.

The server I have is fairly old, so doesn't support NVME, but looks like I could get away with some SSD disks, I am just not sure in my use case if I am wasting my time and they are going to have a very short life (in which case its not worth it). Because its for personal use I am trying to keep costs lower and was looking at something like the Crucial BX500 if it makes sense to use SSD's like this in a ZFS mirror.

I prefer the mirror just for data redundancy sake, I also have off-site backups running but its far easier to get the data from the server than it is to re-download it from cloud backups.

Many thanks!

 

[abufrejoval]

These days SATA SSDs cost more than NVMe drives, so there comes a point where old server hardware may need rethinking.

If it's really server iron with lots of slot space, there are M.2 to PCIe adpaters you might want to use: need to count your pennies for a proper decision.

Unless you really manage giant amounts of constantly changing data, I doubt you need to worry about ZFS or any other RAID eating up your SSDs. And in any case, you can always start with a mirror there, too, where 'write amplification' isn't really an issue.

But if you run things on a server, your network may well be the biggest bottleneck as even a single hard disk can easily saturate a Gigabit link.
2.5 Gbit does only slightly better, each SATA SSD roughly delivers 5Gbit of Ethernet bandwidth, a single NVMe v4/v5 drive might saturate a 100Gbit link.... in theory.

Just saying that you need to balance your setup.

 

[TravisH]

Thanks for your feedback, I appreciate it. Looks like the SSD might be an option, but its an interesting point about the network. I have a bonded link setup but the actual link to the machines is only gigabit so I might be somewhat throwing money away trying to get faster speeds when its over a gigabit network anyway.

I might look at using the fast drive on the end machines, then just running a sync process to sync it to the server and make sure its backed up, then i get the local speed, but have the safety of backed up data.

 

[B.Otto]

The problem with Consumer-grade SSDs is not only the wearout, but the fact that missing PLP prevents them from doing sync writes, which does also result in drastically reduced performance. No matter how good that SSD is in a single-host scenario, once you put it in a Hypervisor it will have to be able to serve a pretty random mix of IOs from several different sources of writes (several VMs, LXCs and copy-on-write file-systems to manage this all).

Thats why everyone recommends buying a used enterprise SSD instead of a new consumer SSD.

 

[Docop2]

As far you keep a bare minimum of 30% of non partitionned disk space and on the 70 that is you do not fill up more than 60, a consumer can do.
When looking at supposed high speed nvme it is only the cache for under 1min, then they all became like a usb3.2 drive at 1gbps. Not to confuse with gigabit network.. this only are 99megs of transfer and the 2.5gb is 236megs.

 

[esi_y]

I wanted to ask about using consumer grade SSD's in a ZFS mirror for home use.

You might want to have a look at this thread here:
https://forum.proxmox.com/threads/2...unt-of-clusterization-how.140434/#post-628788

In my case I have a spinning disk setup with two 16TB drives in a ZFS mirror arrangement. This seems to work well except that as it is spinning disks the speed leaves a bit to be desired.

I am thinking of possibly having another ZFS mirror of a smaller capacity for the stuff I am currently working on (things like photos I might be editing and the likes) where the speed can be much greater than the spinning disks. I am not a professional photographer so I don't think there would be massive file turnover, but I might have 10 - 100GB each time, and every few weeks (at most) I would be putting these on the SSD.

Or just use cache vdev. If you have enough RAM on that system.

I prefer the mirror just for data redundancy sake, I also have off-site backups running but its far easier to get the data from the server than it is to re-download it from cloud backups.

See above.

The problem with Consumer-grade SSDs is not only the wearout, but the fact that missing PLP prevents them from doing sync writes, which does also result in drastically reduced performance. No matter how good that SSD is in a single-host scenario, once you put it in a Hypervisor it will have to be able to serve a pretty random mix of IOs from several different sources of writes (several VMs, LXCs and copy-on-write file-systems to manage this all).

Have a look at 2024 consumer NVMes, find a real-world test, the paper in ancient in terms of what happened to NVMes last few years.

Thats why everyone recommends buying a used enterprise SSD instead of a new consumer SSD.

Not everyone. Not for home use.

 

[Dunuin]

Because its for personal use I am trying to keep costs lower and was looking at something like the Crucial BX500 if it makes sense to use SSD's like this in a ZFS mirror.

Here ZFS killed two BX500 within a year. Wouldn't recommend them. Wouldn't recommend any consumer SSD for ZFS but if you still want to use them, at least get a consumer SSD with TLC NAND. And not crappy QLC NAND like those newer/bigger BX500.

I am thinking of possibly having another ZFS mirror of a smaller capacity for the stuff I am currently working on (things like photos I might be editing and the likes) where the speed can be much greater than the spinning disks. I am not a professional photographer so I don't think there would be massive file turnover, but I might have 10 - 100GB each time, and every few weeks (at most) I would be putting these on the SSD.

The problem is the write amplification. If I write 1TB of data, here it actually writes 20TB to the NAND. A TBW rating of for example 600TB might sound like a lot but with such a write amplification this 600TB would be reached after only writing 30TB.
But it probably won't be that bad in your case if you only want to store photos on it. Biggest problem are small random or sync writes. Sequential async writes are not that bad.

I prefer the mirror just for data redundancy sake, I also have off-site backups running but its far easier to get the data from the server than it is to re-download it from cloud backups.

And without redundancy, there won't be bit rot protection!

And in any case, you can always start with a mirror there, too, where 'write amplification' isn't really an issue.

According to my benchmarks write amplification is more about the workload than the pool layout. Actually the different raidz1/raidz2 3/4/5/6/8 disk combinations I tested resulted in less write amplification than running a 2-disk mirror or 4/6/8-disk striped mirror. As the (striped) mirror is always writing everything twice while a raidz only needs to write a single copy of the data + a fraction of that as additional parity data.
While not great for IOPS performance, that raidz1 was superior for less wear.

But if you run things on a server, your network may well be the biggest bottleneck as even a single hard disk can easily saturate a Gigabit link.
2.5 Gbit does only slightly better, each SATA SSD roughly delivers 5Gbit of Ethernet bandwidth, a single NVMe v4/v5 drive might saturate a 100Gbit link.... in theory.

Fully agree. There are some quite cheap 10Gbit managed Switches and NICs if you are fine with refurbished SFP+ parts. Without upgrading your network those SSDs would be more useful as "special" vdevs to boost the performance of your slow HDDs. Especially when working with smaller files like tons of photos it might be very useful to store all the metadata on the SSDs so listing of folder contents would be very fast.
But especially when working with lots of small files the network bandwidth might not be the thing that is bottlenecking. If you can only read with 1MB/s because the HDDs are already at 100% IOPS utilization then some SSDs would still be great if that would mean bossting it from 1MB/s to the 118MB/s the Gbit LAN can handle.

Not everyone. Not for home use.

It's more about how much you care about your data and what workloads you are running...not if you run your server in a proper datacenter or at home. And yes...I previously also used consumer SSDs at home with ZFS...and I already got a big pile of those consumer SSDs that quickly failed.

 

[esi_y]

Here ZFS killed two BX500 within a year. Wouldn't recommend them. Wouldn't recommend any consumer SSD for ZFS but if you still want to use them, at least get a consumer SSD with TLC NAND. And not crappy QLC NAND like those newer/bigger BX500.

I would also look for at least MX500, but ...

It's more about how much you care about your data and what workloads you are running...not if you run your server in a proper datacenter or at home.

- other than having backups -

And yes...I previously also used consumer SSDs at home with ZFS...and I already got a big pile of those consumer SSDs that quickly failed.

... he does not have a cluster and even with BX, how is the OP going to suffer if he only used it for zcache? It seems his only concern was "speed".

EDIT: I understand this might vary with people, but my understanding of "home use" or "lab" = non-mission critical, definitely no SLA to honour, etc. For disaster recovery, there's the backups, not ZFS.

 

[Dunuin]

EDIT: I understand this might vary with people, but my understanding of "home use" or "lab" = non-mission critical, definitely no SLA to honour, etc. For disaster recovery, there's the backups, not ZFS.

For me, ZFS is part of the disaster prevention. Sure, no raid or snapshots, not even ZFS, could replace proper multiple backups. But in my opinion, without ECC RAM and an integrity-checking filesystem, like ZFS, those backups aren't worth anything. It doesn't matter if I create a backup every hour, store it for 6 months on tapes + in the cloud + offline on rotating drives if I then realize after two years that every 100th file is damaged and that for at least one year. Then all of my backups only contain those identical corrupted files and there is nothing I could restore.
With ECC and ZFS I at least can be sure that all my data is in a healthy consistent state when I back it up.

And yes...I already had non-ECC RAM slowly failing over weeks and silently corrupting hundreds of GBs of data...that happened 3 times at home...since then refuse to buy anything without ECC RAM because in case RAM is failing a 4th time I want to know it instantly before it could cause lots of damage. And yes, I keep my backups for a very long time so I was able to restore 2 months old backups. But I still lost anything I had done the last few weeks as all newer backups of cause also only contained those corrupted files...
And yes, the server storing those data got ECC + ZFS + raid + snapshots...but your data is only as safe as the weakest link...which was the windows PC without ECC that was accessing the data on the server via SMB corrupting every file it opened over weeks...so the best enterprise grade server with all bells and whistles won't help much if you use consumer hardware at home to access it...

And especially those BX500 (TLC version) sooner or later started corrupting data. Not sure if it was worn NAND, a bad firmware or bad controller chip. But every few weeks the ZFS pool was degrading with lots of checksum errors. Wasn't that bad (until both disks completely failed....) because ZFS could repair itself and fix the corrupted data so I don't had to restore some outdated backups. But could have ended really bad when using those disks with a filesystem that is lacking bit rot protection/detection. Then I wouldn't have known that something got corrupted and I would have been happily (over)writing those corrupted files in my backups...

So is your data using PVE with consumer hardware as safe as on your Windows GamingPC? Yes, sure. But the point is that if you have some knowledge of how hardware and software works, a overview over the latest vulnerabilities and you maybe also do some tech support reading all the "Help! All my data is lost!" threads then you realize, that the data isn't as safe as most consumers think it is...

I also once lost 10 years of all family photos/videos. One of my disks failed. I got a backup of that disk on another USB disk. But as I got my replacement disk and was ready to restore the backup, my backup disk also had failed meanwhile and everything was lost as I didn't keek 3 copies of everything at that time...so yes...a single backup isn't enough for home use and you really should follow the 3-2-1 rule...

No problem to use the crappiest Chinese non-name QLC SSD to store your Steam library or to store some torrents...you probably don't care about losing those as you could download them again. But I personally for example wouldn't want to store my home photos/videos on a machine without ECC + ZFS + 3-2-1 backups. I wouldn't even allow client machines to access them (or at least only limited to read-only access) if that client doesn't got ECC RAM.

I had to learn it the hard way that cheaping out on proper enterprise hardware and a good backup strategy isn't worth it if you care about your data...no matter if that is personal data at home or some productive business data...
If I have to decide if I want to spend 200€ for a single conusmer SSD or 2000€ for the same amount of space but with proper redundant servers with ECC and ZFS + onsite backups + offsite backups, I will choose the 2000€ so my data is actually safe.

 

[esi_y]

And yes, the server storing those data got ECC + ZFS + raid + snapshots...but your data is only as safe as the weakest link...which was the windows PC without ECC that was accessing the data on the server via SMB corrupting every file it opened over weeks...

This was almost painful to read, I was originally going to say, for those vouching hard for PLP NVMe's, I wonder if they all also have ECC RAM. The other day there was someone who complained about PVE had IP on the bridge with VMs and their Win VM got ransomware which encrypted their SMB accessible backups. It's really only so far one could go once compromises are made and I think everyone makes them in home setup.

And especially those BX500 (TLC version) sooner or later started corrupting data. Not sure if it was worn NAND, a bad firmware or bad controller chip. But every few weeks the ZFS pool was degrading with lots of checksum errors. Wasn't that bad (until both disks completely failed....) because ZFS could repair itself and fix the corrupted data so I don't had to restore some outdated backups. But could have ended really bad when using those disks with a filesystem that is lacking bit rot protection/detection. Then I wouldn't have known that something got corrupted and I would have been happily (over)writing those corrupted files in my backups...

Fair enough, I do not know BX, I saw e.g. MX500 in use quite a bit though. Samsung PROs, etc. Not the lowest low-end, but I just find the idea of putting a DC grade SSD into (often the case) mini PC funny.

So is your data using PVE with consumer hardware as safe as on your Windows GamingPC? Yes, sure. But the point is that if you have some knowledge of how hardware and software works, a overview over the latest vulnerabilities and you maybe also do some tech support reading all the "Help! All my data is lost!" threads then you realize, that the data isn't as safe as most consumers think it is...

I actually do keep years old backups in AWS Glacier. Now that it's out, waiting for the barrage.

I also once lost 10 years of all family photos/videos. One of my disks failed. I got a backup of that disk on another USB disk. But as I got my replacement disk my backup disk also had failed and everything was lost as I didn't kept 3 copies of everything at that time...so yes...a single backup isn't enough for home use nd you really should follow the 3-2-1 rule...

I have seen e.g. RAID5 having a second drive failing in the very process of rebuilding after failure of the first. It's the empirical experience, I guess, for everyone.

No problem to use the crappiest Chinese non-name QLC SSD to store your Steam library or to store some torrents...you probably don't care about losing those as you could download them again. But I personally for example wouldn't want to store my home photos/videos on a machine without ECC + ZFS + 3-2-1 backups. I wouldn't even allow client machines to access them (or at least limited to read-only) if that client doesn't got ECC RAM.

You are aware this - the last especially - is well above how most people understand home use, though?

I had to learn it the hard way that cheaping out on proper hardware and backup strategy isn't worth it if you care about your data...no matter if that is personal data at home or some productive business data...

Just for the record, I never said that e.g. ECC RAM or PLP SSD is useless, I just find some combinations with certain setups lacking common sense.

 

[Dunuin]

The other day there was someone who complained about PVE had IP on the bridge with VMs and their Win VM got ransomware which encrypted their SMB accessible backups. It's really only so far one could go once compromises are made and I think everyone makes them in home setup.

Ransomware is the one thing I hadn't lost any data to so far. But yes, Ramsomware protection is also a thing you should care about at home and have that covered in your disaster recovery plan. Everyhting important here is snapshotted for 1 year on two different servers where one server is offline most of the time and can't be tempered with. So even with every SMB share being encrypted by ransomware I could revert that for up to a year. Then there is every VM/LXC stored on multiple PBS with proper ransomware protection in place. So no way to delete those backups unless you are somehow able to bruteforce the root passwords of all three PBS servers. And even then there are USB Disks locked away as offline backup as a last resort. If you care about your data, you will have to think about how to recover from such a ransomware attack...even if it only happens at home...

Fair enough, I do not know BX, I saw e.g. MX500 in use quite a bit though. Samsung PROs, etc. Not the lowest low-end, but I just find the idea of putting a DC grade SSD into (often the case) mini PC funny.

I don't see why not. I bought used enterprise SSDs (10 DWPD) with 99% life left for as cheap as 10€. If you are fine with refurbished parts you get a superior SSD for less money than a new consumer SSD. Only downside I could see, putting an enterprise SSD in a MiniPC, would be that those enteprrise SSDs often don't support power safe modes. I for example got a MiniPC with 2x Intel S3710 200GB in a ZFS Mirror where the SSDs are consuming more power at idle than all other hardware together. So you better get something bigger with ECC support in the first place...

I actually do keep years old backups in AWS Glacier. Now that it's out, waiting for the barrage.

Then I'm sure Amazon is capable enough to make sure to not corrupt your data while it is stored there. Still the question if the data already was corrupted locally without you knowing it. Then you have years of backups of corrupted files.

I have seen e.g. RAID5 having a second drive failing in the very process of rebuilding after failure of the first. It's the empirical experience, I guess, for everyone.

I also once (Win98 days I think...) had a HW raid1 array where one disk failed and while rebuilding the array there was a problem and the whole array, including the remaining working disk, was destroyed. So many opportunities to lose data at home.

You are aware this - the last especially - is well above how most people understand home use, though?

I'm well aware what most people run at home. Most people just don't understand how important a proper backup strategy and data integrity are. A friend lost 1 and destroyed 4 smartphones in the last 10 years. Each time she is crying that all photos/videos are lost and I always tell her she should do some backups to not lose everything again the next time it happens....of cause she never learned anything...and all whats left are some pictures uploaded to instagram...
Enought money to spend thousands of euros for all kind of crap but its too expensive to buy a 40€ SSD for backups and too much hassle to connect the smartphone via usb cable once every several months to her PC to make a copy of the pictures...
That most people do it that way at home doesn't mean it is the proper way how to handle this.

Most people also buy crappy and dangerous 5€ USB power supplies from Temu instead of paying 20€ for one that got a real CE/FCC certification so you could be sure it won't electrocute you or burn your house down. And yes, those are often really that dangerous and its horrifying if you watch a video where an electric engineer opens them and explains in which situtions it could easily kill you. That most people use stuff like that at home and that this is "normal" today doesn't make it more safe.

 

[esi_y]

Ransomware is the one thing I hadn't lost any data to so far.

But yes, Ramsomware protection is also a thing you should care about at home and have that covered in your disaster recovery plan. Everyhting important here is snapshotted for 1 year on two different servers where one server is offline most of the time and can't be tempered with. So even with every SMB share being encrypted by ransomware I could revert that for up to a year. Then there is every VM/LXC stored on multiple PBS with proper ransomware protection in place. So no way to delete those backups unless you are somehow able to bruteforce the root passwords of all three PBS servers. And even then there are USB Disks locked away as offline backup as a last resort. If you care about your data, you will have to think about how to recover from such a ransomware attack...even if it only happens at home...

Overwriting backups without testing the new is sort of the problem lots of people make. They have Schroedinger backups then.

I don't see why not. I bought used enterprise SSDs (10 DWPD) with 99% life left for as cheap as 10€. If you are fine with refurbished parts you get a superior SSD for less money than a new consumer SSD.

Some parts I am, some not. I would have no problem with RAM, I would not want HDDs that were spinning 24/7 for 5 years. SSDs I am not sure, it's somewhere inbetween as one can read the TBW %, I can imagine to run them in a mirror if they really cost that little, but that probably happens only when you work in that datacentre?

My issue with PLP SSDs for home use basically was, that in the early days it was cost-prohibitive, nowadays it's different in terms of what's on the market and what's being phased out. I do not know of NVMe M.2s like that, only SATA SSDs and I do care for IOPS (not so much for the rest). In the future what would be looking at? U.2s? Where to plug them? The mini-PCs I know do not have e.g. dual M.2s, definitely not dual SFF bays. And definitely no option for ECC RAM, the best ones are SBCs in fact. That's for the "clustering at home". Now admittedly, I run HP microserver for a fileserver, but no PVE there. That's as little (and quiet) as it gets with ECC RAM, iLO, etc.

Only downside I could see, putting an enterprise SSD in a MiniPC, would be that those enteprrise SSDs often don't support power safe modes. I for example got a MiniPC with 2x Intel S3710 200GB in a ZFS Mirror where the SSDs are consuming more power at idle than all other hardware together

At least it's an SSD and quiet, but that's also the issue that e.g. one does not want the mini PC to overload its power brick from 2 SSDs.

So you better get something bigger with ECC support in the first place...

The issues is that Boeing 747s with no warranty left on them are quite noisy too. Either it's a lab or it's a home *server* which runs 24/7, then it should not be an HVAC all-in-one for the winter.

Then I'm sure Amazon is capable enough to make sure to not corrupt your data while it is stored there. Still the question if the data already was corrupted locally without you knowing it. Then you have years of backups of corrupted files.

I do test them as mentioned, also I was old fashioned in that those were duplicity backups and lots of changed files all of a sudden would have gotten noticed, but I definitely can lose my encryption keys.

I also once (Win98 days I think...) had a HW raid1 array where one disk failed and while rebuilding the array there was a problem and the whole array, including the remaining working disk, was destroyed. So many opportunities to lose data at home.

I think it's also overrated, sure no one likes to lose data, but if we had e.g. a bunch of old photos and a house fire or flooding, they would have been gone as well. It's not a career ending event.

I'm well aware what most people run at home. Most people just don't understand how important a proper backup strategy and data integrity are. A friend lost 1 and destroyed 4 smartphones in the last 10 years. Each time she is crying that all photos/videos are lost and I always tell her she should do some backups to not lose everything again the next time it happens....of cause she never learned anything...and all whats left are some pictures uploaded to instagram...

So here you come to the rescue. But in reality, people do not mind as much as they claim they do. At least it's quite small amount od data volume that's worth keeping backed up to the level of corporate data warehouse. I am sure some professional videographer would be able to disagree, but ...

Enought money to spend thousands of euros for all kind of crap but its too expensive to buy a 40€ SSD for backups and too much hassle to connect the smartphone via usb cable once every several months to her PC to make a copy of the pictures...
That most people do it that way at home doesn't mean it is the proper way how to handle this.

But this is why services like ICloud Photos are probably ... it's sad ... but they are for the most people ... the best solution. And their gaming PC with consumer SSDs.

Most people also buy crappy and dangerous 5€ USB power supplies from Temu instead of paying 20€ for one that got a real CE/FCC certification so you could be sure it won't electrocute you or burn your house down. And yes, those are often really that dangerous and its horrifying if you watch a video where an electric engineer opens them and explains in which situtions it could easily kill you. That most people use stuff like that at home and that this is "normal" today doesn't make it more safe.

This was supposed to be illegal to get imported, I thought. BTW We got too far, I never suggested any of those SSDs. Also, Crucial is Micron, not a fly-by-night operation.

Thanks for the exchange, @Dunuin, and apologies to the OP.

 

[ness1602]

Or don't use zfs,use something that is unsupported like mdadm

 

[esi_y]

Or don't use zfs,use something that is unsupported like mdadm

Better yet, LVM mirror!