SSDs and RAID

[Alessandro 123]

hi to all
how do you manage SSDs in RAID ?
With 2 SSDs in RAID1, they could fail in a very,very,very short time due to the same writes amount on both disks.

Do you make preventive replacements ? If yes, on which parameter you "set" a disk for a replacement?

 

[LnxBil]

Use two different brands of the same size. Always a good idea to install enterprise grade SSDs.

 

[Alessandro 123]

I can't use two different brands. My vendors only provide Intel SSDs (enterprise grade) but I'm reluctant to put two DC3610 in a ZFS mirror (or hardware RAID1, it doesn't change)

They could fail at the same time

 

[LnxBil]

Yes, could be, but not one knows. Enterprise-grade SSD are very stable. We built a lot of SSD-only SANs in the last two years and never had a crash so far. You could swap the SSDs after some time from two servers, but I do not think that you'll write so much, that the disks will fail very soon.

 

[Alessandro 123]

Which kind of RAID do you use in ssd-only SANs ? Same SSD vendor for all disks ?

Yes, I'm thinking to swap disks or make some preventive replacement, but based on which value from smart?

 

[Rhinox]

They could fail at the same time

Yes, they could. But they do not have to. It is not like SSD has given TBW value and it fails just exactly after reaching it. Mostly SSD keeps working long after wear-level parameter indicates it is completely worn out...

 

[Alessandro 123]

Are the TBW a real value or just ipotethical like MTBF?

 

[Rhinox]

Both TBW and MTBF are just informative values...

 

[Alessandro 123]

So, you can't safely depend on these values.
On which smart data we should plan for an SSD preventive replacement ?

 

[Rhinox]

It is up to you. But if you use enterprise-class SSD, I think you replace them due to insufficient capacity/speed long before they reach their lifetime-limit.

For example I think about replacing two sata/SSDs I have been using in PVE-server for a few years. They are still good (with "health" still reported as 99%), but new M.2-PCIe and pure PCIe models are simply much faster...

 

[Alessandro 123]

That's not exactly true.
With SAS disks, I had some servers up and running from about 6 years with the same disks.
I don't think this would be possible with SSDs. SSD tend to suddenly hard-fail, with not "preventive warnings" like reallocations and so on.
With mechanical disks you are able to see that something bad is going on and you have (except some sudden failures) to replace a drive.

If the TBW is just an informative value, how do you detect a replace-needed SSD ?

 

[waltercarroll]

Well, you’re less likely to want to make use of RAID’s performance-enhancing features with SSD, as solid-state drives already provide something like 80 to 100 times the I/O of spinning disk, depending on the usage profile. But you could group a number of solid-state drives together and gain further I/O speed advantage by striping data across them. There are also methods of mixing solid-state and spinning disk to make use of the best characteristics of both, but we’ll come onto that in a while.

 

[LnxBil]

So, we use SSD mostly in RAID10 setups in our SANs, because we do not want to reduce the lifespan (TBW) due to write amplification in RAID5 setups. Our vendors ensured us however that the disks are capable of RAID1 and RAID5 as long as do not exceed the TBW values, which can be easily monitored with smart. The TBW are so extremely high for enterprise class SSDs that they practically do not matter. The wearout is less than 10 percent for over 2 years, so the "normal" life-cyle of 3-5 years is totally fine for the SSDs.

Easiest thing to replace non-working SSD is with a standard support contract with your hardware vendor, e.g. HP, Dell, Fujitsu or IBM. If something breaks, you get a replacement - same as with disks. Yet, we never had a crash (neither SSD nor disk) in the first 2 years.

 

[Alessandro 123]

The point is not how to replace an SSD in warranty. It doesn't matter if the SSD is in warrany or if the vendor is able to replace it quickly. I always have a couple of spare SSDs here so replacement could be made in a couple of minutes.

The point is *when* replace the SSD before it fails. And how to avoid a double SSDs failure in the same RAID-1 set.
If you create a RAID-10, you can loose at least 1 SSD per each mirrored pair. As SSD tend to fails based on the writes, having two (identical) SSDs in RAID-1 means having the same write pattern on both. Thus the probability for a double fail is very, very high as they both waear out in the same way at the same time.

If you look through smart, I almost sure you'll find both SSD (of the same RAID-1 mirror) with the same (or very very similiar) wear out level. This is very risky, because they could fail simultaneously

So, the question is: how safe is to put SSDs in RAID-1 ? Probably, to stay on the safety side, we should replace one of them preventively. Doing so will create a RAID-1 mirror with different wear out level and thus different failure rate. But when to replace these SSDs?

 

[LnxBil]

That's clear, but the disks will normally not fail at the same time if the wearout level is not near or below 0%. It can only happen if the batch of SSDs has a general fault and you cannot rule that out for disks either, so it'll not matter here.

In general: I understand your concern, but in my experience with enterprise SSDs (consumer SSDs is a totally different story), the wearout is the value to go (if you rely on smart). Also the same other smart values apply to SSDs (if present) that applied to disks like error count etc.

 

[Alessandro 123]

Ok, but I think that reported wearout is still an optimistic value computed with statistical data and non a real value.

probably, the wearout is based on DWPD and writted data, but DWPD is statistical, not a real value, thus even the wear out is just informative.

 

[LnxBil]

The wearout is vendor dependent (some count from 0 to 100, others the other way around). Most SSDs I've seen have MB_Written or something like that and that will probably not be statistically guessed, but real measurements.

 

[Alessandro 123]

the mb_written is obviously a real measurement, but compared to what ? Probably they compare to DWPD that is pure statistical, so even the result (like the wearout) is statistical

 

[LnxBil]

You have to ask the vendor for specific questions, I will rely on the measurements the vendor provide and teach in their administration courses.
e.g. the Fujitsu DX systems report exactly the wearout levels back through their GUI to the storage administrator for easy inspection.

From a business point of view, the SSDs are calculated that in their DWPD-lifespan, they will most likely not fail, because you need to replace them and that costs, so if you as a customer stay below this limit ("head flying hours" * mb_written / DWPD) you will probably not have a problem. Monitoring is easy with simple scripts based on the formula above.

I think that at some point, you just have to have faith in the SSDs. If you don't have that (yet), then buy different vendor or different series and combine them. In my experience, this is practically harder if you need to maintain a fully certified system/stack and cannot swap in other SSDs than the ones provided by your vendor.

 

[Alessandro 123]

You have to ask the vendor for specific questions, I will rely on the measurements the vendor provide and teach in their administration courses.
e.g. the Fujitsu DX systems report exactly the wearout levels back through their GUI to the storage administrator for easy inspection.

From a business point of view, the SSDs are calculated that in their DWPD-lifespan, they will most likely not fail, because you need to replace them and that costs, so if you as a customer stay below this limit ("head flying hours" * mb_written / DWPD) you will probably not have a problem. Monitoring is easy with simple scripts based on the formula above.

Yes, but DWPW is an hypothetical value.
Intel DC3710 1.2TB is rated at 24.3PB. By assuming a 520MB/s for writes (that you'll never reach), 24.3PB means:
24.3PB/520MB = 46730769.2 seconds = 540 days

Do you think the vendor tests each SSD for at least 540 days by writing at 520MB/s ? (this would never happen, it's impossible that you'll stay at maximum sequential write speed for 540 days, thus the test time would be much longer)

It's an hypothetical value, and calculations based on hypothetical values only provide hypothetical results.

I think that at some point, you just have to have faith in the SSDs. If you don't have that (yet), then buy different vendor or different series and combine them. In my experience, this is practically harder if you need to maintain a fully certified system/stack and cannot swap in other SSDs than the ones provided by your vendor.

I agree. In fact, I can't use different disk vendors.