[SOLVED] Intel D3-S4520/S4510 vs consumer SSD's like the WD SA500 MX500 Samsung 870 EVO for ZFS VM drives

[VGE]

Hi guys,

So with SSD prices at an al time low. I am planning to upgrade my Proxmox server so I can build more VM's and learn more about databases and monitoring.

Current setup:

• 2x Kingston SA400 120GB ZFS Mirror just for Proxmox. No VM's, LXC etc.
• 2x Samsung 850 evo 500 GB ZFS mirror for VM's, LXC etc.
• 3x 2TB WD RED RAIDZ1 for Bulk storage

The two SA400, I got for free. So for now perfect for just the OS. The Samsung's are old and are not enough space so I am looking for an upgrade to about 2TB SSD's.

My consumer options are:

• Crucial MX500 2TB - € 104,89
• Samsung 870 Evo 2TB - € 118,99
• WD Red SA500 2TB € 130,66

My other options are:

• Intel D3-S4520 1.92TB - € 185,28
• Intel D3-S4510 1,92TB- € 185,48

I read a lot about enterprise SSD's being a lot better for ZFS, but are they € 50,- a piece better? I have a fully tested backup strategy to a separate server and a second backup to an offsite backup so I don't need extreme loss protection.

The consumer drives do state a lot higher write IOPS about 88.000 vs 38.000. But like 15 times the TBW/DWPD.

Would really like to hear thoughts and input! Don't mind to spend some extra cash to get a lot better hardware!

 

[wallacio]

Ask yourself if the cost of spending time recovering from data or disk loss is greater or less than spending the extra $50/$100 on more reliable, better performing hardware. For me, it's a no brainer - even before I start counting the cost of my sanity and the welfare of others around me when the poo hits the fan

Spend the extra cash.

 

[LnxBil]

You need to compare if the drives are QLC, TLC, MLC or SLC. This is the main difference between them. The main point in speed is the write performance and enterprise SSDs are much better in writing small blocks randomly and have also a much lower write amplification. Consumer SSD have a much higher TBW with small changes than enterprise SSDs depite their potentially higher DWPD. If you write a 1 MB block for any 4 KB change for consumer and 8 KB block for any 4 KB change for enterprise, the difference is already factor 128 fold which is much higher than 15.

I've been running samsung enterprise SSD for more than 10 years with server load and they have not reached 10% wearout, yet I reached 50% wearout with ZFS on a samsung 750 pro in 2 month.

 

[Dunuin]

The consumer drives do state a lot higher write IOPS about 88.000 vs 38.000. But like 15 times the TBW/DWPD.

And when doing sync writes the IOPS will drop from 88.000 to something like 400 as they don't got a power-loss protection to be able to cache sync writes. Enterprise SSD might not got such a high burst performance but at least the performance is more constant and won't drop that bad after writing a couple of seconds...

For me the max performance is unimportant. All I care about is the min performance that you always get in all situations (DBs doing 8K sync writes, continously writing over hours, ...) and here a enterprise SSD is magnitudes faster.

Even if a consumer PCIe 4.0 NVMe SSD is advertised with with 4000MB/s of write performance, it can only handle that under specific workloads (like big sequential parallel async writes) and only for a few seconds. Hit it with unparallelized random 4K sync writes and it will drop down to something like 2MB/s. While a crappy SATA enterprise SSD with only 200MB/s advertized write performance might still write with 40MB/s.

 

[VGE]

@wallacio I thought some of those consumer SSD's have PLP but after your comment I checked and they don't or its not same protection as Ent. SSD's. Thats a good point and def worth 50 bucks imo. For the product itself, do you know if the D3-S4520/10 are good enough Ent. drives?

@LnxBil all the drives im compairing are TLC, also the two Intel D3-S4520/10. Are the Samsung drives you are referring to TLC or better? Is there a way to check the write performance online for these drives? Or is that the TLC/MLC/SLC comparison you are referring to?

@Dunuin is there a way to check trough a test or spec-sheet if the SSDs im looking at (Intel D3-S4520/10) have a bigger/enough cache sync writes? Yea with my new plans I will be running a lot more DB's. Good to know, I never focused on continues write speed. Need to read in on it.

 

[Dunuin]

@Dunuin is there a way to check trough a test or spec-sheet if the SSDs im looking at (Intel D3-S4520/10) have a bigger/enough cache sync writes?

Its not about the cache size (but enterprise SSDs usually got a bigger cache too. They use the same DRAM cache that is used for async writes. The difference is the power-loss protection. With it they can cache async + sync writes in the DRAM cache, without it only async writes, as all cached data in the volatile DRAM will be lost otherwise.

 

[santiagobiali]

Your enterprise options are way better than the consumer ones. If it's for homelab, I would even consider buying the Intel used if they still have 90%+ TBW remaining.

Consumer grade SSDs can't keep up with the IOPs.

 

[VGE]

@Dunuin Thanks. That makes it a lot more clear. Im going for the Intel ones. Maby a side question, do you suggest 1 drive with good backups or a mirror for safety and better read speeds? Or maby a stripe?

@santiagobiali thanks for the reply. I made my decision im going for new intel drives. Used are € 125 a piece and its not really that I don't have the € 180 but more, should I spent it. Im convinced I need the Intel.

 

[Dunuin]

@Dunuin Thanks. That makes it a lot more clear. Im going for the Intel ones. Maby a side question, do you suggest 1 drive with good backups or a mirror for safety and better read speeds? Or maby a stripe?

Mirror for bit rot protection + not wasting time setting everything up from scratch when a SSD fails + no downtime + no lost data since your last backup.
And raid doesn't replace a backup. You still should have 2 other copies of everything with one of them offsite.
So ideally a mirror + a onsite + a offsite PBS.

 

[LnxBil]

Is there a way to check the write performance online for these drives?

There is/was this site that had some data, no idea if it gets updated.

 

[VGE]

@Dunuin sorry maby stupid question but isn't ZFS already taking care of the bit rot? I have a 3:2:1 backup strategy. In my home: the VM + Backup to NAS + NAS Backs up to external drive and the NAS makes a full copy of all changes to an offsite backup.

@LnxBil Thanks, great tip! The Intels seems to perform pretty good.

 

[Dunuin]

@Dunuin sorry maby stupid question but isn't ZFS already taking care of the bit rot?

You need mirrors/parity data for bit rot protection. Without it, when using a raid0 or single disk, it can only detect corrupted blocks/records but can't fix them, as there is no healthy copy or parity data to caculate a healty version (= bit rot detection only). So with a mirror it will automatically detect and fix data corruption. Without a mirror it will just complain what blocks/files got corrupted and you better have a 1+ months (= older than your last scrub) old backup of that so you can restore a healty copy.